Ted Joyce
Director, Zicklin’s Online Learning and Evaluation
This past fall, on Sunday, December 20, faculty teaching Principles of Microeconomics (E1001) gave a common final to all 819 students still enrolled in the class. Faculty in all eight sections agreed to use a common text with the publisher’s software. We augmented the publisher’s material with professionally developed videos which we downloaded for free and with our own less professionally produced but better targeted videos. All the material was seamlessly integrated into the publisher’s platform. All sections were responsible to cover the same set of chapters. With this core structure, faculty were free to conduct classes as they saw fit. One section was run as a flipped classroom (Professor 1) and another was offered completely online (Professor 8). The remaining six sections were traditional face-to-face classes that met twice a week for 75 minutes each.
Despite the uniformity of content, there were large differences in student performance by section on the final. Table 1 shows the unadjusted and adjusted mean percentage point differences in scores on the final by Professor. The reference category is Professor 1.
The range is dramatic. Students in five of the seven sections scored at least 13 percentage points less on a scale of 0 to 100 than students in Professor 1’s class. Some of this difference can be attributed to differences in student characteristics by class. When I statistically adjust for these characteristics using linear regression, the mean differences fall by between 3 and 6 percentage points and the adjusted R2, or the percent of the variation in test scores explained by the model, rises from 13.8 percent to 35.7 percent. Thus, the mean difference between Professor 6’s students and those of Professor 1’s students falls from 15.1 percentage points before adjustment to 9.30 percentage point with adjustment.
Statistical adjustment matters because there are substantive differences in student characteristics by section that affect performance. For instance, students in Professor 1’s class had the highest average GPA (3.40) just prior to the class and the highest Math SAT (624) among the 8 sections (see Table 2 below). Both variables are highly predictive of student success in ECO 1001.
Six of the eight instructors were adjuncts. However, before you say, “Well of course the adjuncts would not be as effective at the tenure-track or tenured faculty,” please note the following:
1. All six adjuncts had taught E1001 at least once before.
2. All six adjuncts taught in the traditional format meaning they had at least double the face-to-face time with students compared to the two permanent faculty.
3. Even when we compare only the adjunct Professors, large differences remain. Table 3 below shows the unadjusted and adjusted mean differences in performance by adjunct Professor. The student in Professor’s 2 class scores almost 8 percentage points less than those in Professor 5’s class.
4. We standardized the support of adjuncts across sections. They all received instruction on how to use the software and material was shared across sections. For instance, we videotaped a two-hour live review session for the final exam. The review session went over each problem from the previous year’s final exam. The video was edited and posted within 24 hours and made available to the students in all sections. We also made available printed copies of the final exams from fall of 2013 and 2014 along with detailed answers. The final given on December 20 consisted of 40 multiple choice questions that were created by senior faculty. The adjuncts did not participate in the construction of the final. The questions were similar in style and difficulty to those from the two previous finals that had been posted and reviewed.
Lessons
1. A favorite management expression is, “If you cannot measure it, you cannot manage it.” We don’t measure student learning well. We need more uniform instruments and a common final is a start. This is especially true in the large introductory classes with some of the most appropriate being ECO 1001, 2002; STAT 2000; FIN 3000; ACC 2101, ACC 2203; MKT 3000; MGT3121; BUS 1000.
2. But faculty resist common finals. Yes it takes some organization. Yes faculty must agree on a common text, material, etc., but this should not be difficult in basic classes. We should be able to say, “Here are key concepts students need to know. Here is a test that assesses their grasp of these core ideas.” Even if the entire test is not the same across sections, there could be a subset of questions across all sections.
3. A riff on an old boxing expression is: “It’s not how much time you spend in class, but how you spend the time.” I have observed many instructors, adjuncts and full-timers alike. Many do not use class time effectively. Reading from slides, writing too much on the whiteboard, and not calling on the full range of students are common practices that fail to optimize scarce class time. There is growing literature on how students learn, but a shared insight is that active learning strategies are more effective than the passive lectures. This does not mean 70 minutes of class discussion, but rather structured exercises in which students work with the instructor and other students to solve problems and answer questions in a supportive but serious environment.
4. How faculty help students learn core concepts is where differences in student performance arise. But in order to address those differences we need to eliminate variations in texts, material covered, and exams. The idea is not to embarrass faculty whose students are underachieving, but to iterate towards practices that improve learning.
5. There are many reasons why faculty resist common finals in the basic introductory classes. “I prefer a specific textbook; my exams are different; I teach different concepts; it is a faculty’s prerogative to decide what is important and how to teach it.” Each of these comments prevents an objective assessment of student learning and insures poor supervision of adjuncts who teach upwards of 50 percent of our classes. As the results from ECO 1001 demonstrate, there can be large variation in student learning even with a standardized curriculum. Without a common instrument with which to assess learning, we cannot begin to narrow these differences.
6. Organizing a common final takes some effort. You need to create a common exam and avoid having faculty competitively teach to the test. You must schedule a time to hold the exam. You need a common grading system. You need to adjust for differences in student abilities as measured by GPA and SATs. But all this is doable if faculty are willing.
7. We at ZOLE, Zicklin’s Online Learning and Evaluation, are here to help faculty organize and execute a common final. We will also assist or even do the evaluation. There is much to be gained in terms of improving student learning by working collectively as a faculty.
8. As much research has shown, helping teachers become more effective is not difficult. Getting faculty to engage in this process is much harder. Great teachers may be gifted, but good teachers are made.