Filed Under: Zicklin Online Learning and Evaluation February 6, 2017
by TJoyce
Ted Joyce
Director, Zicklin’s Online Learning and Evaluation
On a Sunday in the Fall of 2016 faculty teaching Principles of Microeconomics (E1001) gave a common final to 868 students enrolled in the course. Faculty in all sections used the same text and online platform and all agreed to cover the same chapters. Four of the ten sections were run as hybrids that met once a week for 75 minutes; three sections were traditional classes that met twice a week for 75 minutes each and two traditional sections met once a week for 2.5 hours. Faculty used the publisher’s online platform to varying degrees but, in general, the hybrid sections required more online work than the traditional lectures.
Class size varied between 40 and 114 students. All hybrid sections had between 79 and 114 students. Four of the five hybrids were taught by Ph.D. students and one by a newly hired lecturer in her first semester. Two of the traditional sections were taught by tenured faculty.
To create the final exam each of the ten faculty submitted 15 questions to a senior faculty who was not teaching the class. The senior faculty selected 40 questions insuring contribution from each participating faculty.
Despite the uniformity of content, there were large differences in student performance on the final exam by section. Table 1 shows the unadjusted and adjusted mean differences in scores on the final by professor and format. The coefficients are the mean percentage-point differences on the final exam between each listed class and the reference class, which was a hybrid section whose students had performed the best. Adjustment included controls for students’ GPA, SAT scores, cumulative credits, gender, and class status. [1]
The results were noteworthy on several counts. First, students in the three traditional classes that met twice a week (highlighted in blue) scored the lowest among all 10 sections. All three differences were statistically significant. Even after adjustment for student characteristics, students in the three traditional classes that met twice a week scored between 4 and 10 percentage points lower on the final exam relative to students in the most successful hybrid class. The other point is that adjustment eliminated differences between the hybrid sections with one exception (Prof10). In general, students in the hybrid classes scored significantly better than students in the traditional classes.
Why were adjuncts teaching the hybrid section so successful? One explanation is that the adjuncts followed the course developed by a senior faculty who was not teaching that semester. The course made extensive use of online instructional material. Faculty in the hybrid classes required that students complete online exercises every week. A more detailed analysis is needed to test this conjecture, but the requirement of extensive online work appears to be a key difference between the hybrids and the traditional classes.
The results underscore the need for common finals in large introductory classes. This is the only way to assess pedagogical innovations, supervise adjuncts, and improve student outcomes.
[1] We thank John Choonoo of the Office of Institutional Research for baseline data on student characteristics.
