Motivation
I recently sat in the back of a large lecture hall during a class given by a talented Professor. It should not have been so jarring. I also teach large lecture classes and yet I am so consumed by my own presentation and interactions, that I don’t fully absorb the varied levels of student attention. But sitting in the weeds of a large lecture hall, one sees class differently. Students’ attention drifts in and out of class. They are checking their phones, computers, and iPads. And even if a student doesn’t have a computer to look at, the beautiful graphics on the laptop next to her have to be distracting. We could insist on no phones, laptops, iPads, etc., but in a large lecture class in which many students make legitimate use of these devices (for at least some of the class), banning them would appear a losing battle.
And maybe their distraction is justified. Perhaps they have seen the powerpoint slides and know when to let their mind leave. Perhaps they figure on reading the slides later. Perhaps the lecture or lecturer is boring. Because it is unlikely that anyone is going to call on them to say anything, the temptation to drift in and out of lecture is overwhelming given students’ access to the world outside. How many faculty have looked down at their iPhones during a long meeting in large room?
So why do we insist on so many lectures if the educational yield may be so low? Could students do without class twice a week? Could they learn the core ideas on their own and then come to class to fill in the gaps? Can we trust them to be so disciplined? How can we get answers to these questions? The response, of course, is that we should study the issue.
This is the motivation for the randomized controlled trial my colleagues and I are conducting at Baruch (see the interview with Ted Joyce ). In a nutshell, we are testing the performance of students in a hybrid versus a traditional lecture format of Introductory Microeconomics. In the traditional format, students meet twice a week for lecture and once a week for recitation. In our version of the hybrid format, students have lecture once a week with no recitation. Instead of the latter, we have videotaped 14 recitations in which a I go over 10 multiple choice problems each week that are similar to ones they will see on the exam. In addition, students have a computed graded online quiz that must be completed before each lecture as well as post-lecture quiz several days after lecture. The quizzes force students into the material and help to keep them from falling behind. We have made the online quizzes and videos available to all students in both the traditional and hybrid format. Our goal to is test a simple question: Do students who have class once a week perform as well as students who meet twice a week all else equal?
Running an RCT
The simplicity of the question belies the challenge of conducting a randomized experiment with students. The first major hurdle is approval from the Institutional Review Board (IRB). The IRB requires a rigorous presentation of the research design. This includes a detailed description of how students are recruited, statistical power, and the benefits/risks to students from participation. Once IRB approval is obtained, the next step is recruitment and consent. With the Registrar’s unwavering help, we sent all potential students information about the experiment. Once they had registered, we followed up with an email asking them to consent. Working with Vice-President Corpus and his staff person, Amanda Nardi, we created a website where students could electronically agree to be in the study. We worked with Elaine Cataletto and staff from the Center for Academic Advisement Center so they could adequately counsel students about participating in the study. In short, we received invaluable assistance from all sectors of the Baruch Community.
Why Randomize?
Key to the study is the randomization of students between the hybrid and traditional formats. To illustrate, assume instead that students select the format of their choosing. Confident learners choose the hybrid; those working many hours also choose the convenience of a hybrid; more risk adverse students choose the traditional format. Because of purposeful sorting, it becomes difficult to assess whether differences in performance between the two formats is driven by differences in the students or differences in the format. All this is solved if students are successfully randomized. By assigning students to a format with the “flip of a coin” we insure that students of similar ability, working hours and preferences are distributed equally between formats. Thus, if we observe a difference in test scores between the two formats, we can be confident it is due to how the class is taught and not the students in the class).
Initial data
The table below shows characteristics of students participating in our study. Consider cumulative GPA. The average GPA is 3.04 in the traditional class and 3.05 in the hybrid. The difference is statistically zero. The same holds for all the other characteristics. In more technical terms, participants are balanced between the two study arms. This is key to the study and it took many months and a whole “village “ of Baruch staff to accomplish.
What will we learn?
Suppose students in the hybrid class do as well as students in the traditional class. The implications are potentially profound. We may no longer need to schedule two lectures a week in the introductory classes of finance, marketing, management or accountancy which appears popular with students (see the interviews with students from the class). This will free up scarce lecture hall space. It also will enable us to service more students as fewer faculty will be needed at the introductory level enabling more faculty to service higher level classes. But this is only one scenario. What if we find that only students with a GPA of 3.0 or higher in the hybrid class did as well as their counterparts in the traditional class. This would suggest we limit hybrid class by student
We intend to publish our findings in an academic journal. We expect there to be great interest in our results. College Presidents, Provosts and Deans are pushing their administrations and faculty to convert courses from traditional formats to hybrids, pure online and even MOOCs. Despite the huge interest in the these new formats, there is a paucity of evidence as to their impact onf student performance. There have been few rigorous studies as to what works and what doesn’t. We hope to begin to fill that gap.
We will have a lot more to share after the semester.
- Ted Joyce is a Professor of Economics in the Department of Economics & Finance in the Zicklin School of Business.