Analytics for Class Lectures

The classes for the new academic year have started, so naturally I started thinking about teaching-related topics.

Mining video interactions

A few days back, FXPal released TalkMiner, a system for indexing and searching video of lecture broadcasts. One of the interesting ideas is that it is possible to mine the interactions of students with the video, to see what are the topics of interest for the students, what parts of the class get skipped, and so on. From the blog post of FXPal:

The Berkeley webcasting system (developed by our president Larry Rowe while he was a professor there) showed that

… students almost always watched the lectures on-demand rather than in real-time, and they rarely watched the entire lecture.  Students use the webcasts to study for exams – we could see this clearly by patterns of usage – and, they primarily wanted to review selected material covered by the instructor.  In one class we discovered that for over 50% of the lectures, students watched less than 10 minutes from a 50-minute lecture and students watched the entire lecture only 10% of the time.  Consequently, for using the system, effective search is a big issue.

At Stern, all the classes get recorded and are available to students for reviewing the class material. The students get access to a layout like the following and have the ability to rearrange the layout, emphasizing the slides, or the video. (You can see a lecture of mine; login: scribe and password: Scribe987!)

It seems to be a natural next step to show to the instructor the patterns of interaction that students have with the videos. It would be very interested to see what parts of the class go largely unexamined and which ones are played again and again. Needless to say, these are either complicated topics, or topics that the instructor did not explain clearly.

Mining search queries using transcripts

Another interesting idea is to also have transcripts of the class. (For example, for this lecture [login: scribe and password: Scribe987!] see the transcript, done by CastingWords for $0.75/min.) This would allow students to search the class not only using text in the slides but also to recall particular points of the class discussion. This is especially important for courses that have a significant component of in-class discussion. We already know, from web search, that query logs are important source of information. Doing the same for class content would easily identify what students are looking for in the class recordings.

One problem with transcription is that it is rather expensive. CastingWords and SpeakerText seem to charge one or two dollars per minute for human-verified transcriptions. (Fully-automatic solutions are not ready for prime time, as the automatic transcription of these YouTube videos shows. Make sure to click the "cc" button and then "transcribe audio".) With approximately 28 lectures a semester, 75 minutes each, at 1-2 dollars per minute, we have a cost of $2000 to $4000 per semester. At this cost level, it is certainly more beneficial to hire an extra TA rather than provide the transcription of the lecture to the students.

Mining class participation 

Another thing that I would love to have is the ability to transcribe not only what the instructor said but also who are the students that contributed to the discussion, together with what they said. This would allow not only to track and quantify participation but also uncover some patterns that may not be obvious to the instructor.

For example, take a look at this diagram below, created as part of the yearly teaching evaluation that we undergo at Stern:

The diagram was created by an evaluator who sat in my class, tracked the composition of the student body, where each student was sitting in the amphitheater, how many times they raised their hand, and how many times I asked them to answer a question. (To answer the inevitable question: No, the teaching feedback is not focused only on such analyses. In my earlier years, the feedback was focused more on substantial issues, e.g., structuring lectures and discussions, encouraging participation, etc. Now, with feedback and experience, the more substantial and important issues are addressed.  So we focus on such, seemingly more superficial, but also important, stuff...)

The results? I was paying significantly more attention to the left part of the amphitheater: I asked 80% of the time students sitting in the left, and only 20% of the time I asked students on the right. Also, the percentage of female students participating in the discussion was significantly lower: 50% for male students participated, but only 21% of the female students did.

These are patterns that are hard to understand while teaching, but would be easier to find out if we had detailed transcripts of the class discussion together, potentially, with a standardized seat chart. I was also told that some universities (the rumor is about Harvard Business School) use software to track student participation. However, I was not able to locate any such software offerings. 

Moving forward

The ability to videotape lectures has been around for a while and is being used extensively for distance learning applications. (Columbia Engineering had a well-established distance learning program when I joined the PhD program back in 1999.) However, it was mainly a broadcast mechanism, and not a medium for providing feedback to the instructor (and even to the students who can see that they are lacking in terms of participation). 

It would be interesting to start having such technologies for providing feedback on teaching. Analytics have been changing many industries. Education has been surprisingly behind in that respect.