Tuesday, September 20, 2016

Flipped classroom? No thanks, I'd rather you lecture at me

As an early adopter and enthusiast of MOOCs, I've followed the "flipped classrooms" and "lecture is dead" conversations with some interest. Indeed, why would students attend lecture—and why would professors repeat last semester's lectures—when high-quality versions are available online for viewing anytime?

This semester, I'm once again teaching Software Engineering at Berkeley to about 180 undergraduates. (Past enrollments have been as high as 240, but we capped it lower this semester to make projects more manageable.) In the past, Dave Patterson and I have team-taught this class and we've developed a fairly well-polished set of lectures and other materials that are available as a MOOC on edX, Agile Development Using Ruby on Rails, and which we also use as a SPOC.

Last spring, by the end of the semester our lecture attendance had plummeted to about 15% of enrollment. We surveyed the students anonymously to ask how we could make lecture time more valuable. A majority of students suggested that since they could watch pre-recorded lectures, why not use lecture time to do more live coding demos and work through concrete examples? In other words, a classic "flipped lecture" scenario.

So this semester I dove in feet-first to do just that. Less than 20% of my lecture time has been spent covering material already in the recorded lectures; instead, we made those available to students from the get-go. The rest has consisted of live demos showing the concepts in action, and activities involving extensive peer learning, of which I'm a huge fan. I've even started archiving the demo "scripts" and starter code for the benefit of other instructors. The "contract" was that we would list which videos (and/or corresponding book sections) students should read or watch before lecture, and then during the lecture time (90 minutes twice a week) the students would be well prepared to understand the demo and ask good questions as it went along.

I told the students I would also sprinkle "micro-quizzes" into the lectures to spot-check that they were indeed reading/viewing the preparatory materials. The micro-quiz questions were intended to be simple recall questions that you'd have no trouble answering if you skimmed the preparatory material for the main ideas. (We use iClicker devices that are registered to the students, so we can map both responses and participation during lecture to individual students.)

Today in lecture, a bit more than 4 weeks into the course, I've officially declared the flipped experiment a failure. (Well, not a failure. A negative result is interesting. But it's not the result I expected.)

Since we made the pre-recorded lectures available within an edX  using the edX platform itself, and students have to login with their Berkeley credentials to access the course, we can see using edX Insights (part of the platform's built-in analytics) how many people are watching the videos.

According to the edX platform's analytics, the typical video is partially viewed by about 20 people. Only 45 people have ever watched any video to completion. Remember, this is in a class of 180 enrolled students, whose previous cohort specifically requested this format.

Maybe people are watching the videos after lecture rather than before? If they were, you'd expect video viewing numbers to be higher for videos corresponding to previous weeks, but they're not.

Maybe people are reading the book instead? If they were, the performance on the microquizzes should be nearly unimodal—they are simple recall questions that you cannot help but get right if you even skim the video or the book—but in fact the answer distributions are in some cases nearly uniform.

Maybe people already know the material from (e.g.) an internship? One or two students did approach me after today's lecture to tell me that. I believe them, and I know there are some students like this. But we also gave a diagnostic quiz at the beginning of the semester, and based on its results, very few people match this description.

Maybe students don't have time to watch videos or read the book before lecture? This is a 4-unit course, which means students should nominally be spending 12 hours per week total on it, of which lecture and section together account for only 4. The reading assignments, which can be done instead of watching the videos, average out to 15-20 pages twice a week, or 30-40 pages per week. Speaking as a faculty member leading an upper-division course in the world's best CS department at the world's best public university, I don't believe 30-40 pages of reading per week per course is onerous. Also, in past offerings of the course, we've polled students towards the end of the course to ask how many hours they are actually spending per week. The average has consistently been 8-10 hours, even towards the end where the big projects come due. So by the students' own self-reporting, there's 2-4 hours available to either do the readings or watch the videos.

As you might imagine, planning and executing live demos requires a couple of hours of preparation per hour of lecture to come up with a demo "script", stage code that can be copy-pasted so students don't have to watch me type every keystroke, ensure the examples run in such a way as to illustrate the correct points or techniques, remember what to emphasize in peer-learning questions at each step of the demo, and so on. But it's discouraging to do this if at most 1/9 of the students are doing the basic prep work that will enable them to get substantive value out of the live demo examples.

So at the end of lecture today I informed the students that I wouldn't use this format any longer—those who wanted to work through examples could come to office hours, which have been quiet so far—and I asked them to vote using the clickers on how future lectures should be conducted:

(A) Deliver traditional lectures that closely match the pre-recorded ones
(B) Cancel one lecture each week, replacing it with open office hours (in addition to existing office hours)
(C) Cancel both lectures each week, replacing both with open office hours (ditto)
(D) I have another suggestion, which I am posting right now on Piazza with tag #lecture
(E) I don’t care/I’ll probably ignore lecture regardless of format

Below are the results of that poll. The people have spoken. Less work for me, but disappointing on many levels, and I feel bad for the 20-30 people who do prepare and who have been getting a lot of value out of the more involved examples in "lecture" that go beyond the book or videos. And it doesn't seem like a great use of my time to do a live performance of material that's already recorded and on which I'm unlikely to improve, though it is a lot less work for me (if not as interesting).

(Note that the number of poll respondents adds up to 121, consistent with previous lectures this semester. So even in steady state at the beginning of the semester, 1/3 of the students aren't showing up, even though participation in peer-instruction activities using the clickers counts for part of the grade.)

(Update: I posted a link to this article on the Berkeley CS Facebook page, and many students have commented. One particularly interesting comment comes from a student who took my class previously: “When I took your class, as someone who rarely, if ever, went to lecture or prepared, the microquizzes helped me convince myself to do that. They weren't hard, so getting them wrong was just embarrassing to me. I was probably the best student I ever was until you stopped doing them my semester for reasons I can't recall and after that the whole thing fell apart for me.” So maybe I should just stick to my guns on the "read before lecture" part and make the microquizzes worth more than the ~10% they're currently worth...)

A year or so ago, I approached the campus Committee on Curriculum and Instruction to ask whether they'd be likely to approve a "lab" version of this course, in which there is no live lecture (only pre-recorded), the lecture hours are replaced with open lab/office hours, and all the other course elements (teaching assistants, small sections, office hours for teaching staff, etc.) are preserved. They said that would likely be fine. It's sounding like a better and better deal to me given that the majority of students want me to spend my time doing something nearly identical to what I've done in past semesters. And if previous semesters are any indication—and this has been very consistent since I started teaching this course in 2012—lecture attendance will fall off about halfway through. (I won't be surprised if the rationale given is that the lectures are available to watch online, even though the data we're gathering this semester shows that's clearly not happening.)

In an ideal universe, maybe we'd have 2 versions of the course, one tailored to people who do the prep work and want a fast-paced in-depth experience in lecture and another for people who prefer to be lectured to in the traditional manner. But we don't live in that ideal universe because of finite resource constraints. And one could argue that we already have that second version of the course—it's on edX and is the highest-grossing and one of the most popular courses on the platform, and it's free.

Instructors: What has your experience been flipping a large class?

Students: What would you do?


  1. You've shown that the students being taught either way said that they'd prefer the other style of lecture. To me, this says that they'd like to have a reason to go to lecture, but in their current format it's not worth their time. Unfortunately neither lecture formats appeared to be worth the majority of the students' time. Again though, the students don't vote for "choice E" in the in-class poll, but rather for the other lecture format, which tells me that they'd like to come to lecture if they'd get something out of it.

    My opinion: most students do close to the minimum they feel they need to do to get the grade they want, and right now they can get by without lecture. They can pass, and even get A's by doing the projects and homeworks, cherry-picking things out of the book/lectures when they get stuck.

    1. As a student I have to say it depends on the subject. Lecture on useless information that you can reference and memorize - ignore. Lecture on skills you need for testing and assignments - pay total attention.

      A big problem is though, when information is taught but skills are vague. Like in my Python class: "Here's how iteration and loops work. Now write a program doing something complicated".

      And it becomes, "I know I have the tools to do this, I know how to use these tools, I just can't figure out how to have them work in concert for this goal".

      So, I dunno, maybe it just depends on the class and the interests of the students; but there's responsibility on both sides..............................

  2. Until this semester, there was a version of CS 61 A at Berkeley that was taught exactly in the way you propose. It was canceled, unfortunately, for reasons that are not clear to me. I took it last year and found it incredibly useful and an excellent way to organize the class. I would love it if this format was used.

    1. It was canceled because of a lack of TAs to teach the course :(

  3. I posted this in the Berkeley Facebook group, but thought I would share here:

    The goal is to help people learn, right? In that case, it's not clear to me that a lecture is the best method for everybody -- but I do think it's a pretty good one for many students. (Also depends on the topic).

    Nothing (digital) compares to the experience of a good live lecture. Note that the word good is important in the sentence.

    Good means two things, which I think you already know:
    * The lecturer has to be engaging. I'm not actually sure if there is any data that correlates engagement to learning, but it does seem to contribute the the desire of students to attend course. I can't recall off hand any papers about this, but I do know that there are plenty of examples of lecture attendence differing widely simply by varying the sage on the stage. In general, I think I found CS169 lectures engaging.
    * The lecture has to promote active learning. I think the µquizzes are a necessary component for this -- regardless of whether they are graded, but they aren't the only component. Breaking lectures into chunks, and talking with peers are also useful -- even if a number of people still don't do the talking.

    The challenge seems to be having people keep pace with the course in a non-stressful manner. To be honest, I'm not sure that this is super possible given two things:
    * berkeley students -- because 103% of us are oversubscribed.
    * edx - there's not a good way (IMO) to check on progress and to check on the little things that get done. To many deadlines and policies are hard to manage.

    Personally, I've never been a big fan of live coding demos. They can be useful, but often times I find them hard to follow (no reference material if needed), and always at the wrong speed. (I.e. I don't need to see someone typing, but I do need to spend time contemplating what was executed. These times are nearly always in the wrong propritions for live demos.)

    The model that I've been thinking about is some weekly or even almost daily practice to keep up with course material. Perhaps you could fullfill this requirement by having µquizzes in lecture, but allowing students to substitute them for something of similar difficulty online.

    In the end, I think the way to handle stress to have relatively lenient course policies. Naturally some students will take advantage, but I don't think most do. In the end, if students are more stressed out or too busy, they're probably not learning the material anyway.

  4. I went to a class with a reversed classroom approach on computer architecture
    at Uppsala univeristy in Sweden and got an overall good impression.
    The students attending this course were a grab-bag of science majors (as opposed to CompSci majors)
    so the tempo was a bit too low and the depth too shallow for my taste.
    That being said, the video interface had buttons and keybindings for upping the tempo of the videos
    and skipping a few seconds in either direction.
    This allowed me to listen to slow paced content at a reasonable pace and I could skip the already familar material.
    I am very thankful of this feature, boredom can be minimized.

    Each lecture was posted online with accompanying problems
    and was followed by an in-the-flesh problem solving session.
    These sessions were done by (sometimes) thinking about
    the problem yourself for a few minutes and then (always) discussing it with a partner.
    The teacher would wander among us and see how we were doing and help if a pair got stuck.
    Each problem took a couple of minutes to solve.

    Each lecture had quiz questions followed by the answer and an explanation interleaved with new material.
    To pass the course you had to view the lecture and answer the quiz questions before the problem session.
    I think this is super helpful to procrastinators like myself,
    doom is then always imminent if you do not study in advance.
    This may be the ingredient you are lacking.
    To not study in advance in a course with just a final exam as a deadline is almost as dooming,
    just less apparently so.
    Sure, cramming will _maybe_ let you pass but your grade and more importantly, your understanding will suffer.
    A great incentive structure for the procrastinator.

    Also, of course you can skip through the lectures and answer the question randomly and continue without looking at it or the explanation.
    But this is also tedious and if you go and open the lecture video you are a tiny step from studying, so you might as well, right?

    There were a few optional lectures which were not covered by the exams.
    I think a few in-depth and/or out-of-scope optional lectures is a good idea, I wanted more of them.
    This is a way for the educator to point out relevant material and keep the interested student well fed.

    That's the good things I want to highlight.
    There were also a few bad things that could have been avoided.

    The web interface for the lectures took like 20-40 seconds to load due to
    some backend or infrastructure issue on opencourseware's side.
    The interface was only designed in chrome, it looked a bit off in firefox.

    The exam was a multiple choice answer with 72 questions.
    Pro: They can test every topic of the course and grading is fast and simple.
    Con: The time spent (by the professor and TAs) constructing each question is lower
    which lead to badly or ambiguosly worded questions and answers.
    Adding insult to injury, it is really hard to make up four plausible sounding alternatives to a question.
    Some answers on the exam previous years were even wrong.
    I don't know if this is an inherent tradeoff in making lots of multiple choice questions
    or if it is possible to accumulate, refine and recycle old exam questions after numerous exams.
    My guess is the latter, but until then, a few (in my case maybe 2-4 out of 72) exam questions will suffer.

    I hope you found something useful in all of this.
    If you want to trade notes with my professor regarding this, here is his university profile with an email address:


  5. I think this is a great blog post because it highlights some of the real issues with the flipped class approach. I wouldn’t call what you have done here a failure, I would call it an early bail-out data point, and I think it’s a really common experience for people who try flipping a classroom without enough support. We have been really fully embracing the flipped class/active learning model in computing classes at UNC Charlotte (including a class which is fully flipped and ‘wrapped’ around your MOOC). We are having a lot of success with this model, but many of us have made mistakes along the way, and your experience sounds like what some of our faculty have experienced the first time they tried to do some of this. I’d like to elaborate here on a few points that I think highlight why your experience might have been less than satisfactory.

    First, running a flipped class for only four weeks is simply not long enough for students to get into the rhythm of prep work and active learning. This is especially true if the students have never done a flipped class before. Your class is likely the only flipped active learning class that many of the students have experienced, so it goes against their expectations.

    It doesn’t sound like there were enough forcing functions to ensure students did the prep work. In our classes, we typically have homework quizzes that students have to take before coming to class. These quizzes test whether students did the assigned reading and watched the assigned videos, and they are worth 10-20% of the students’ final grade. We often set them so that they ‘close’ 30 minutes before class starts. This helps to motivate the majority of students to actually do the prep work before coming to class and it means the faculty member doesn’t feel like they have to spend time explaining what was in the videos or readings.

    You also asked students what they wanted, which is very different from which way they learn the most. Students will typically want you to teach in the way that requires the least effort from them, but this also typically means that they don't learn much. The standard ‘sage-on-the-stage’ delivering a PowerPoint lecture requires very little active effort from students, and it’s often what they are used to and what they expect, and therefore what they want. But the research shows that this isn’t the way to help student actually learn material.

    We have found that we have to explain to the students (over and over!) why we are flipping the classroom and why we are forcing them to engage in active learning activities. We cite the research and explain to them why these active learning activities are good for their brains, information retention, skill gains and knowledge synthesis. We also now have the benefit of students who have taken three or four of their first computing classes as flipped, active learning classes. We are starting to see these students demand that more of their classes be taught this way, because we are changing their expectations around what university education is supposed to be. In the world of MOOCs, our active learning flipped classrooms are one of the answers to the question: What is the added value of an in-person university education?

    Finally, creating active learning materials is a LOT of work. But, if you actually repeat this class next semester and get a chance to amortize the time spent developing the active learning materials by re-using them, you may feel more positive about the experience. I have spent hundreds of hours creating countless short videos, and hands-on peer instruction and active learning activities for my classes, but I re-use them over and over, and then I get the fun of going to class and actively engaging with the students, instead of standing behind a podium, boring them to death with PowerPoint slides.

    Don't lose hope, Armando - this really can be so much more rewarding, if you give it a chance!

  6. My experience of flipping a large class is...that you should take your First Attempt In Learning how to do it, reflect on it, and try different strategies next time.

    You make a lot of good points in this post, I don't want to invalidate them. But trying something once, for four weeks, is not really a commitment to trialing a new pedagogy. I've been sent this article THREE TIMES today by people trying to use it to "prove" that flipped learning "doesn't work".

    Tell that to TEDtalks.

    Tell that to YouTube.

    It may be the case that you are an excellent speaker and that your material is best suited to live lectures. Or that your cohort (who may be mostly full time enrolled, non-parents etc) is able to attend and engage in lectures. That doesn't mean that flipped learning "doesn't work". It means that your lectures "do work". Many do not. In fact many are awful.

    Can I suggest two things if you embark on this for a second cycle:
    1. Don't record lecture-long videos. 20 minutes is the maximum TED Talk length for a reason, and the production value of those is high. YouTube teaching videos are often shorter, 10 min chunks at most.
    2. Don't try and do 'workshop style' activities with giant lecture halls full of people. The point of flipped learning was for students to use class time for teacher/peer contact instead of content transmission - you'll never achieve this unless you get a smaller teacher:student ratio.

    I love a good live lecture, don't get me wrong. They suit the way I like to learn. But there's more than one way to skin a cat.

    1. Posting on behalf of Armando:

      Kelli, thanks for your reply.

      I definitely appreciate your thoughtful comments, but I want to make sure you have the correct context around what we did, and what I do and don't claim about the experience.

      In particular, nowhere do I say "flipped learning doesn't work", and whoever is using my post to "prove" that statement is putting words in my mouth. I have seen flipped learning work, and in models more compelling than TED talks. What I said was *this* experiment hasn't worked, and I'm trying to determine why. I've taught this class for four years, in brick-and-mortar and online, in crash-course format, to both college students and professionals/continuing ed students, in cohort sizes ranging from 55 to 240. We have tried many experiments. Some have worked better than others. But this was not an uninformed experiment undertaken without context and lightly abandoned, as your reply seems to suggest.

      I appreciate the suggestion to not record lecture-length videos. In fact, the pre-recorded videos to which the post refers are 6-15 minute segments created and curated over a 3-year period in conjunction with a MOOC, and each includes one or more self-check questions, as the de facto "MOOC best practices" suggest (in fact I think our course was the first or second course ever on both Coursera and edX, and we remember "discovering" some of these things along with Andrew, Daphne, and Anant). Berkeley does record the full-length screencasts of live lectures and makes them available to students, but that is IN ADDITION to the curated and pretty well-organized short videos that we made available in this case. (The reason to make the full-length ones available is precisely that if I do a demo or other livecoding activity in lieu of traditional lecture, it's a way for students to review that or catch up if they missed lecture.)

      I also agree that "workshop style" activities work poorly in a huge lecture hall, which is why we haven't done them. We do peer instruction using clickers and small group discussion, which works quite well (and has worked since before we thought about flipping), and I try to do livecoding activities and demos that invite some level of student participation and make it easy for them to "follow along" on their own computer (not sure if that counts as a "workshop style" activity or not). Indeed, we're trying to figure out what mix of activities could work with a cohort this large.