Yes, according to a study conducted by David N. Figlio, Mark Rush, and Lu Yin, “Is It Live or Is It Internet? Experimental Estimates of the Effects of Online Instruction on Student Learning” (National Bureau of Economic Research, June 2010). I’m tempted to say “of course,” but for reasons other than those intended by the authors.
Subjects and Procedure: “Students in a large introductory microeconomics course at a major research university were randomly assigned to live lectures versus watching these same lectures in an internet setting, where all other factors (e.g., instruction, supplemental materials) were the same” (abstract).
Conclusion: “Our strongest findings in favor of live instruction are for the relatively low-achieving students, male students, and Hispanic students. These are precisely the students who are more likely to populate the less selective universities and community colleges. These students may well be disadvantaged by the movement to online education and, to the extent that it is the less selective institutions and community colleges that are most fully embracing online education, inadvertently they may be harming a significant portion of their student body” (21).
Unfortunately, based on this report, many will use this argument — in-person live lectures are superior because they are more effective with a wider range of students — to defend traditional live lectures against the encroachment of online learning.
But there are a number of flaws in this study that caution against such a logical leap, and perhaps the most glaring is the use of video recordings of the live lectures in the online treatment. This goes against the grain of conventional wisdom. The literature is saturated with admonitions against transferring F2F (face-to-face) pedagogy into the VLE (virtual learning environment). This imposition of methodology suitable for one medium into a completely different medium is high up on the list of worst practices.
If “low-achieving students, male students, and Hispanic students” don’t take very well to video lectures, it doesn’t necessarily mean that they won’t do well in online courses that don’t rely solely on recorded lectures. There are other ways to deliver the information contained in live lectures, and online instructors are aware of them. Text-only or audio-only versions of the lecture are options. The audio could be replayed via an iPod during a jog, on the bus, or while snacking. The text version could be read anywhere, anytime via an iPad or netbook — in a fraction of the time it would take to view the video.
If the online treatment offered students choices, the affected subpopulation might have fared better. For example, the lecture could have been posted in three different formats: text, audio, and video. Given a choice, would low-achieving, male, Hispanic students choose the video? They might, and they might follow up with the text or audio version to ensure mastery. Or they might not and simply choose to read the text version or listen to the audio.
Despite this and other flaws, though, this study is worth reviewing as a resource for further research. The authors have taken the time to discuss the limitations of their investigation and implications for further studies.
Filed under: Uncategorized | Tagged: David N. Figlio, Experimental Estimates of the Effects of Online Instruction on Student Learning, Is It Live or Is It Internet, Lu Yin, Mark Rush, National Bureau of Economic Research, NBER |
John Mark Walker: "If educational communities can continue to push platform integration and content portability, in the future, students may be able to design their own personalized degrees from smaller, modular chunks that cross institutional barriers" (
Richard Koubek
Judith McDaniel
Tim Fraser-Bumatay: "Although the format leaves us far-removed physically, the online forum has its own sense of intimacy" (Judith McDaniel, "
Ryan Kelly: "For me to be able to work with people clear across the country for an extended period of time opened me up to new things" (Judith McDaniel, "
Daniel Herrera: "As a Mexican American, I know that words of identity are powerful; so to discuss white privilege with my professor and classmates in a face-to-face class would have been terrifying and impossible" (Judith McDaniel, "
Cathy Gunn: "Traditional methods for effecting change at my institution aren’t getting us even to a trickle yet, let alone to thinking about or planning for a wave!" (
Educational Technology and Change Journal 
Jim,
Thanks for the post. Very interesting. I agree with your critique. An additional question would be, is a video lecture better than NO lecture. Astonishing numbers of students in large classes do not attend live lectures, even when no substitute is available online. (My basis for this statement is anecdotal. I remember being thrilled to be allowed to collect data during the first 10 minutes of a 200 person live course. Showed up with 200 surveys, but only about 50 students showed up for class.) Does anyone know of hard data about lecture attendance?
In the experiment, exposure to either live lecture or video was controlled. In the real world, what percent of students who skip live lecture make up for doing so by watching online later?
Carrie
Is it really reasonable to look only at video, audio, and/or text as the alternatives to live lectures, which as Carrie so neatly commented, few attend anyway?
Online opens up a host of alternatives that this study does not even begin to address. Blended choices bring even more options into play.
I have to wonder, not being an education college being, why this study was even done. Although the detail about some groups faring poorly has interest, we should not be focusing on small differences here. We should be looking for giant leaps forward given the technological resources available.
Video lectures are just SO last century.
Great questions, Carrie. Re lecture attendance — a few years ago, I recall visiting an award-winning smart classroom in Southern California and asking the staff member, who served as our guide, about attendance when much of the course info is also provided online. He stated that attendance could be a problem but (wink-wink) they kept a step ahead of the students by making critical info available only in the in-person lectures.
This same sort of comment was made by a professor at a prestigious college. He was appalled at the poor scores students received on exams that tested their ability to recall key points in his lecture. The context for this comment was the growing “problem” of students multitasking (going online with netbooks) during his lectures.
In both cases, instructors have placed their delivery preference over learning. If the purpose is to learn, then wouldn’t the teacher want to expand and facilitate students’ access to critical information — rather than artificially narrowing the access to channels of his/her own choosing?
If the prof felt that certain bits of info in his lectures were vital, then he could have also listed them in a text file and made them available online. If his next argument is that the students would stop coming to his lectures if he did that, then perhaps he’s not hearing the message that the students are sending him.
-Jim S
Harry: “Is it really reasonable to look only at video, audio, and/or text as the alternatives to live lectures, which as Carrie so neatly commented, few attend anyway? Online opens up a host of alternatives that this study does not even begin to address. Blended choices bring even more options into play.”
Harry, you’re absolutely right! Video, audio, and text are only the tip of the iceberg when it comes to online or blended — or even F2F — options. There’s a whole bunch of discovery, exploratory, active, etc. approaches that could effectively replace the lecture.
As you say, the lecture as the standard for instruction is long gone. The new standard has to include a healthy dose of online tech used in ways that place a premium on the learner as an active and integral part of the learning process.
-Jim S
The authors also specify that the course in which they conducted the experiment in offered a very rich online learning environment: students of both experimental groups had access to it, the only difference being in the enforcement in the 2 experimental groups of, respectively “F2F only” and “online only” modalities for the lectures.
For the “F2F only” students, access to the videos of the lectures was disabled in their accounts, but they could still use the other resources; the “video-only” students were prevented from entering the lecture hall (the students who did not participate in the experiment actually alternated between both modalities, even though they were theoretically registered for one or the other).
The authors also admit that some “F2F only” students may have by-passed the video block by using a friend’s account. Hence the tentative nature of their report. The reason why they published it is that in a savings-driven shift to online teaching, there is a risk that other institutions with lesser means may limit it to putting recordings of lectures in an LMS, without providing the same rich online learning environment.
Their hypothesis that some “video-only” students performed less well at exams because they attempted to cram video study till the very end is interesting. Their proposed remedy – forcing students to download/watch the recordings within some days of the lecture – less so: you’d have to use DRM for that, and DRM is anti-educational – and eminently by-passable.
Jim’s suggestion of also offering the lectures in audio and text format is much better. But transcribing a lecture takes time and hence money – unless you make the transcription part of the course assignments as was the case for Charles and Rebecca Nesson’s Fall 2006 CyberOne course at Harvard (1). After each lecture, one or several students had to publish its transcript or their notes about it in the wiki for the course.
And maybe, as long as university students come from a school system where learning is still prevalently based on text, universities should introduce them to studying on other media. Listening to audio recordings of lectures with an iPod, as Jim suggests, may work for some, but others, especially non native speakers, may be better off with an audio editing software like Audacity, where you can navigate very precisely, slow down the pace while keeping the pitch, and take notes in the “labels” track.
(1) The recordings of the CyberOne lectures were offered both in audio and video – again, in an otherwise rich online learning environment. The course was opened to registered in-class and distance students, and to everyone interested.
Claude, thank you very much for the clarifications and explanations! They add much to the article and the discussion. Your comment re Audacity is especially useful. I can imagine students who are native speakers of English, too, finding this tool useful!
Re the CyberOne lectures — I’m definitely in Harry’s camp re looking for alternatives to lectures. The ultimate learning-as-interaction question may be: How can we involve, engage the student in both the creation and learning of course material?
Perhaps one way is to get away from content as predetermined answers to cutting edge issues in the field and turning to the issues themselves. For example, wouldn’t it be great if, say, an econ course were based on a series of issues or topics in the form of questions — instead of prof-determined “answers” presented as content?
And discussions could focus on different responses to the questions, with the prof massaging the give and take so that a wide range of relevant as well as not so relevant views are shared.
The prof’s role would be that of an impartial panel host, and the students would serve as panelists. Part of the learning task would be to locate reliable sources on the web. Students could be encouraged to experiment with various media in presenting their views.
The prof would strive toward keeping the discussions open as long as possible, forestalling closure and consensus. Student reports would present positions that also appreciated the complexity of the issue.
This is beginning to sound more like a grad course, but maybe that’s a clue, too. I hated lecture classes — large and small. But I loved upper division and grad classes where the profs were more interested in what we had to say than their own prepared lecture notes.
In the give and take, time flew, and the best profs always had us on the edge of our wits, guiding us toward clarifying our own theories then inserting an anomaly that blew it to pieces, forcing us to rethink our ideas. These profs rewarded students for taking theoretical risks, for thinking outside the envelope, and they made learning exciting and fun. Writing papers for them was an extension of the discussions. And students were, again, rewarded for thinking rather than spewing back the prof’s lecture notes.
-Jim S
I have not been able to access the full study. I filled out the form with my university email and received an onscreen confirmation that my university status qualifies me for access to the report. Supposedly it was to be emailed. It’s been hours and still no email. Looking at your comments, it seems as if Claude and Harry have been able to read it… Any suggestions what i should do?
Thanks!
Carrie
Carrie, I just emailed you an attached copy. Let me know if you don’t get it. -Jim S
Thanks, now I have read the manuscript. I have some added concern about the case the authors make about male, Hispanic, and low achievers performing significantly less well online than live. It is hard to tell from the data, but it looks as if there are about 8 Hispanics in the LIVE group. No detail is provided about the low achievers. Test scores for these groups are not reported in the tables, though the text mentions that differences were 2 points or less. The major finding of the study is that there were no significant differences between online and live lecture study participants. Too much, and at the same time, not enough, is made of these shadow differences.
The media will pick up on them, not the main finding.
We do not know what the lectures were like (except, apparently, questions were not allowed to be asked, I guess, otherwise that would have been discussed as an experimental difference between the conditions…). Were there informative graphics? Did they use something like camtasia to show graphics and the instructor’s face? Is the instructor a good lecturer?
Still, good for them for conducting a careful study.
Re Carrie Heeter’s “The media will pick up on them, not the main finding.” In fact that’s what THE Chronicle did – see Video Lectures May Slightly Hurt Student Performance by Sophia Li, which was modified and whose original title was changed on June 22 after it elicited an avalanche of abuse against the researchers in the comments, “to more accurately characterize the research.”. I didn’t see the original version, but the revised one remains bad enough.
It’s a bit as when the Pope discussed a text by Manuel II Paleologus during a seminar at Ratisbon university in 2006: he was talking as a theologian to theologians, forgetting that being also a pope, journalists were present: they predictably reported the quote from Paleologos out of context and caused an uproar among Muslims. See the full text of his speech (in German) on the Vatican’s site.
Now. the pope being a pope also means that there is a full Wikipedia article – http://en.wikipedia.org/wiki/Regensburg_lecture where the media reports are referenced and analyzed, thus putting the whole affair in perspective. This is not likely to happen with the abuse hurled at the authors of this report, so thanks to Jim for reopening the discussion here.
Re how the lectures were recorded: being able to properly see visual aids used by the lecturer is essential, of course. Yet in the CyberOne course I mentioned before, Charles Nesson’s lectures were filmed in one shot from a fixed camera and it was fine because he is a great speaker, uses few visuals, and anyway there were links to further materials in the pages for the lectures. These pages – to answer Jim’s and Harry’s remarks about finding alternatives to lectures – also linked to assignments, e.g. creating video games about the theme treated ;-)
But CyberOne is a Harvard course and Harvard has money, probably more than the university where David N. Figlio, Mark Rush, and Lu Yin did their experiment. Not that any part of the rich learning environment for CyberOne required costly software: even the Berkman Center island in Second Life used – e.g. – for tutorials for the distance students of CyberOne, was apparently sponsored by Linden Labs. What is expensive is the time needed to conceive a rich learning environment and effective, creative online activities.
Carrie, the questions you raise in your comment are excellent.
Your comment re the results, though, might need some clarification. You said that “The major finding of the study is that there were no significant differences between online and live lecture study participants. Too much, and at the same time, not enough, is made of these shadow differences.”
You’re right. There were no significant differences in the “Comparison of Average Test Scores for Live Versus Online Instruction.”
However, there were significant differences in “Heterogeneous Effects of Live Instruction Versus Online Instruction” for three subgroups:
Hispanic students 11.276*** (.01 level of significance)
Male students 3.480** (.05)
Low-achievers 4.054*** (.01)
The dependent variable is the average test score.
-Jim S
A mathematician’s viewpoint on the use of stats in human sciences: Tom Lehrer’s Sociology
;-)
More seriously, now: RSA Animate – The Secret Powers of Time, an animation of a conference by Philip Zimbardo: contentwise pertinent to the time management issue mentioned in the report, and in its form, a pointer to alternative ways to use recorded lectures for studying. Some simultaneous interpreters say that they go from source to target languages via a kind of mental cartoon.
Claude, nice videos!
Thanks, Jim.
But I find Table 4 to be mystifying.
Hispanic students 11.276*** (.01 level of significance)
Male students 3.480** (.05)
Low-achievers 4.054*** (.01)
The caption says: Dependent variable is the average test score measured on a 0-to-100 point scale. Number of observations: 296.
It appears that 8% of the live group and 12.6% of the online group were Hispanic (based on earlier tables). There were 3 exams, does the dependent variable add up all 3 exams and the divide by 3 for a single composite score or did they pick just one of the three exams? It seems like the Hispanic comparison looks at the mean differencd between the 8 live Hispanic students and the 27 online Hispanic students? It seems rare in my experience for mean differences in an N of 35 to ever be significant. How can the sample size for this test be 295? What statistical test was used? Very confusing.
Did the poor performing Hispanics in the online group happen to also be male and low performing? Might this really be one difference reported 3 ways?
Carrie, thanks for the clarification! Good questions. I hadn’t delved that deep into the data and simply accepted the results. I should’ve followed up on your earlier comments re this data (in the reply that I referred to) before I posted my reply. I hope the authors read your comment above and respond. -Jim S
Claude, the first video is hilarious! It gives those of us who rely on stats a moment to look at ourselves. Probability as a means to determine significance of outcomes is a useful tool, I think, but when it’s applied to poorly designed studies to invoke credibility, it does, as Lehrer suggests, become hokey.
I found the second fascinating in terms of presentation. Yes, it definitely appears to be an entertaining and effective alternative to straight lecturing — going a step better than cartoons for message delivery.
What bothers me, though, is the message that digital devices are the cause of present orientation (hedonistic, fated/deterministic) and, consequently, poor academic performance in especially male students. Perhaps this is the shortcoming of Zimbardo’s approach — it may not work with complex issues.
Thanks for sharing these.
-Jim S
[…] https://etcjournal.com/2010/06/28/4656/ […]