Complication: Respondus does not work with all types of technologies that students might use to take a class. 

Resulting questions:

• Respondus does not work on all technology platforms, so how does one equitably ask students to use it? What if only some of the students in class have the technology that is compatible? As of April 2020:
  • Respondus works well on iOS and Windows but does not operate on the Chromebook/Android operating system (note that Baruch distributed Chromebooks as loaner devices to students in need during the ‘recalibration’ period).
  • Respondus Lockdown will work on Chromebooks if it is used in a “kiosk” mode, but Respondus Monitor does not function in this mode. 
• If students do not have a webcam attached to their home computer or refuse to switch on their webcam for this purpose, they will be unable to complete this assessment.  

Complication: To work effectively, students must have access to reliable, high-speed internet for the duration of each exam monitored by Respondus.

Resulting questions:

• How do we accommodate students who do not have this kind of internet access in their homes?
• Respondus’ own documentation suggests that students should ask other users sharing the same internet connection to go offline while they take their exam. Is this realistic in households with more than one member who may have time-sensitive commitments online?
• If an internet service interruption occurs while a student is using Respondus the program may cut off and not allow the student to reenter. How can we assess the validity of a student’s claim that this has happened? How do we give re-tests to students who need it because of internet service interruptions?

Complication: Respondus Monitor relies on recording students where they are taking their exam (most likely in their homes). 

Resulting questions:

• How do we address privacy concerns? What happens if a student is taking an exam and there are other people in the background (such as minors)? It is illegal to record minors without parental consent.
• Many of our students have housing insecurity, which may worsen given the current crisis. If a student is taking this exam in a place where they are unable to control the lighting conditions or other environmental factors, their exam footage could be marked as “suspicious” by proctoring software, prompting the faculty member to review it and violating students’ privacy. What if a student is taking the exam from a shelter?
• Recordings are stored with Respondus for 5 years. These recordings are also available for this long (perhaps unnecessarily) to the instructor via Blackboard. Beyond this who else has access to these recordings?

Complication: Use of APS raises important information privacy issues. Adoption of such systems may contradict what we teach our students about information privacy issues. 

Resulting questions:

• Do students have control over who has access to information recorded over the duration of their exams? Would it be sold to any third parties?
• Is it a fair choice for students when they have to decide between withdrawing from a course vs. consenting to the use of intrusive technologies? 
• What message does it send our students in terms of our values and priorities when it comes to the privacy of their information?

Complication: Behaviors that Respondus flags as suspicious (i.e. facial tics or expressions that might be considered inappropriate to context, involuntary movement of the hands or body) have a higher propensity to unfairly single out students with disabilities and/or students who can’t control their environment. 

Resulting questions:

• Baruch’s mission highlights our diversity and inclusivity. How do we navigate these values while using a technology that is based on a more homogeneous set of norms?
• Respondus Monitor AI states that it tracks student facial and body movement, the lighting in their environment, and the ways they interact with their technology—including mouse movements—“to identify patterns and anomalies associated with cheating.” These patterns are built on student data, and student data collected by the system is fed back in turn (their privacy statement indicates that they can use video and audio collected from student exams) which creates a troubling cycle of reinforcement around behaviors that have been deemed suspicious.
• How would the college recognize and address any issues of algorithmic bias? That is, how could the college know that suspicious behavior flagged automatically by such automated proctoring systems are not based on a parameter other than actual suspicious behavior? To what extent do various factors (such as test data, models) contribute to potential discrimination? This is especially important as Baruch has a diverse student body (along race, socio-economic status, etc.) and if the vendor has primarily provided solutions to institutions that are very different from Baruch, the predictions could be error-prone. For example, facial recognition software is worse at recognizing people of color.

Complication: What is the flexibility with students who need accommodations? 

Resulting questions:

• ADA compliance requires that there be alternative proctoring available. What would that look like? 
• What does an opt-out option for students look like? How can students feel empowered to opt-out without feeling like they are endangering their grades?

Complication: Despite making it more difficult in some ways for students to cheat, there exist online forums that detail how to workaround Respondus.

Resulting questions:

• Respondus Monitor uses photo ID to establish identification. Unfortunately, it cannot detect a well done fake ID, even one on paper or cardboard (as opposed to the plastic ones printed by the college). How can we be sure students aren’t using fake IDs to link their name to an alternative test-taker?
• How effective is Respondus at ensuring academic integrity when there are already online forums that offer advice such as on Reddit and YouTube?

Emphasize Academic Integrity 

Make sure you communicate your values and expectations when it comes to academic integrity in your course. For example, in some courses it is expected for students to collaborate with each other and share notes and resources (often in group projects) and in others it is considered cheating. Please don’t assume that the norms for test taking are exactly the same in each class.        

• Provide students with a statement about your expectations of their test-taking conduct when it comes to academic integrity and honesty. (For example, clarify if it’s okay for them to refer to notes or google a fact.) This is often best accompanied by a statement specifying the steps you will take as the instructor to assess their work for plagiarism (e.g., making use of online software, such as safeAssign). 
• Begin each exam with your expectations regarding academic integrity and include a non-graded pre-question about it. 
• Ask students to write their own short agreement to uphold academic integrity and submit it to you in advance of the assessment. For example, “I pledge to not make any use of my textbook or other unauthorized sources as I complete any part of this test.” Signed: student’s first and last name.” 
• Alternatively, require students to answer T/F to the statement about upholding your academic integrity expectations and being responsible to do their part. 
• Communicate your values, and be open to feedback. Rather than simply reminding your students not to cheat, encourage a dialogue with them about why they do before it well before it happens. You might ask students to read a brief summary of research about the motivations of cheating to open a discussion about your own expectations, and how you can make assignments more meaningful, accommodating, and transparently assessed.
• Look at how other faculty discuss academic integrity at Baruch College. It includes examples of the kind of language you can use in crafting statements about academic integrity and honesty, including those that can be incorporated into your syllabus.

Reconsider your Exam Design

Explore different types of exam questions. 

• Consider giving open-ended assessments (e.g., short answers, short essays, writing-through-the-process responses), given constraints such as class size. In this series on issues in education (part 1 and part 2, via informs.org), Prof. Will Millhiser from Zicklin explains the value of creating open-ended questions for an introductory management science class. 
• Ask students to offer their “take” on an issue, topic, or concept (e.g., describing relevant personal examples, or ways to apply the content in a unique way), or ask them how to explain how they might arrive at a solution that you would otherwise have them calculate.  
• Develop ‘backwards multiple-choice’ tests. Give students a set of concepts and terms and have them create multiple choice questions based on those concepts. 
• Ask students to pick the correct answer from a multiple-choice test and also explain why they picked that answer (can give partial credit for a reasonable explanation even if what they picked is incorrect.)
• Share the test itself—untimed—and encourage students to use outside resources. Have them document the sources they consulted to find the answers. 

Rethink how you create and set up the exam on Blackboard.  

Faculty are encouraged to visit Teach Hybrid to learn more and find links to BCTC’s tutorials. Please consult with BCTC with questions regarding Blackboard. 

• Display questions to students only one at a time.
• Prevent backtracking and randomize the order of questions. 
• If using multiple choice, consider also randomizing the response options. 
• Create large pools of test questions and tests that randomly draw from these pools. Faculty can create algorithmically generated questions wherein values used for certain parameters could be automatically generated. 
• Add an overall time frame to the test so that it is only available for completion during a particular period. For example, make your assessment available for students on Thursday from 4:15pm to 7:15pm or from Friday midnight until Monday morning at 9am. Please be mindful that courses have pre-assigned slots for final exams. Going outside those time periods risks all sorts of conflicts. Even during the semester, extending outside of class time can conflict with other classes, not to mention other student obligations.Please keep in mind that the time frame will have to be doubled for a student with a disability who has extended time. 
  • One upside to having a shorter time frame (e.g., Thursday from 4:15pm to 7:15pm) is that it minimizes chances students will be dishonest by screenshotting test items and sending them around to other students.
  • One downside of a shorter time frame is that students may not have availability or accessibility during that particular window of time. For example, what if wifi goes down in a neighborhood that Thursday late afternoon and students can’t access the test? Have a back-plan to offer a make-ups exam available.
• Add a timer to the test, so that once students start the test they would only have a certain amount of time to complete it. However, it’s important to remember that some of your students might need accommodations and are entitled to have a longer time limit. You should reach out to the Student Disability Services with any questions. [Students are required to register with Disability Services, show their “Accommodation Card” to the faculty, and discuss those accommodations with the faculty. This is not up to the faculty to initiate. Prior to an exam, the student should communicate their accommodations to the instructor.  
• Choose whether you allow students only a single attempt at completing the exam or multiple attempts. The drawback of choosing “single attempt” is that students can have legitimate technical issues that disconnect them from the test.

Offer an open note or open book exam. 

Send your exam to students, and give them a specified amount of time in which they can complete it.  Include a statement regarding your expectations for academic integrity. As part of the exam, ask students to confirm that they have read and abide by your definition of academic integrity. 

Alternatives to Exams

When it comes to assessing student learning, there are a number of reasonable and effective alternatives to tests and quizzes. Consider replacing your exam with the following suggestions compiled through our own Hybrid Seminar and TeachHybrid work, as well as borrowed from Baruch’s Writing Center and the CTL at UC Berkeley: 

• Assign a written response (a memo, an annotated bibliography, a presentation, a case analysis, etc.) that asks students to demonstrate mastery of the same concepts you typically access via the timed test. See examples below:
• Topic (or textbook chapter) “summary” or “briefing” paper: Students select a single topic from among any number that you provide (e.g., a main section in a textbook chapter; a novel theory; a current issue). 
• Annotation of a themed set of contents: students develop a thematic and organized annotation of several key pieces of works, writings, readings, research, and/or sources of information. 
• Reflective (or expressive) journal entries: students express their thoughts, feelings, and/or opinions in a reflective piece or journal entry (single or repeated) affording them a unique opportunity to think about how the concepts they are learning about apply meaningfully to their own lives and/or communities. 
• Would-be op-ed piece: students weigh two (or all) sides of a key issue in the course – and to do so in a real-world manner with an Op-Ed piece. Asking students to take on such a challenge requires that they first demonstrate mastery of related content, and then think about how best to write the piece for public consumption (or some other preferred, directed audience).
• Portfolio: students document their independent work, or revise a series of assignments that they have completed throughout the semester.
• You may borrow from and/or get inspired by the CTL’s TeachHybrid’s repository of hybrid assignments, or this list of innovative hybrid assignments from CUNY faculty across various disciplines. Note that some of these may not be a good fit for fully-online classes, but they can be retooled with a bit of imagination.

For a more detailed explanation of the above strategies, check out the Quizzes, Tests, and Alternative Forms of Assessment blog post on the TeachHybrid site, and scroll down to the section on Alternative Assignments. 

Alternatives to Final Presentations

Have students present in small groups—either in writing, via video-conference, or via uploaded recorded video/audio. Give them a guide for responding to / assessing each others’ presentations, and have them submit these responses to you after they “meet.” 

Additional Considerations For Future Semesters

With the recognition that this document is addressing examination strategies for an emergency teaching moment, we also wanted to include some suggestions for rethinking the role of assessment in the anticipation of preparing for distance learning over the summer.
 

The CTL encourages faculty to weigh whether the exam can help students to demonstrate a skill along with content knowledge. In many work and civic environments outside of the classroom, students will be encouraged to collaborate with other people, to locate reliable information, and / or to engage in creative problem solving outside of a strictly timed and monitored environment. Consider how experiential learning and project-based learning might help your students to accomplish the course learning objectives while also developing key collaboration and curation skills.

Transparency

Specify that you will be using Respondus and the technology requirements under the notes in CUNYFirst so that students know in advance that this is part of the course. Send an email to students after they register for your course (this can be done as a mass email once a week) to ensure they are aware in advance of relevant policies for your course. Also mention these things upfront in your syllabus so that students know what they are signing up for. It’s a general “best practice” to describe on CUNYFirst and in your syllabus any tools and approaches you may use in your course that might raise concern among students.

Practice with a less intrusive, lower stakes assignment first.

Experiment with Respondus on the least intrusive assignments first, and in lower stakes assessments before attempting to use it for a high-stakes final exam. Or let students “play-test” in a mock-test so that they can become acquainted with the software, your guidelines, and the behaviors that are appropriate for test-taking. Trying these approaches  well before any main or larger-scale assessment is to occur will help you and students address any challenges and prepare contingency measures should certain elements of such technologies fail. Also, if you choose to use Respondus Monitor, providing Respondus LockDown Browser as an option for students who refuse to use their webcam during an exam may provide some flexibility. 

Accessibility considerations

According to Student Disability Services, Respondus is accessible with the JAWS screen reader for PC and the Voice-over screen reader for MAC and iPad. Accessibility is automatically enabled for all users, no adjustments need to be made by faculty. For “Speech to Text” with Dragon Naturally Speaking, users have reported adjusting the settings to “not use the dictation box” in Dragon Naturally Speaking, which will allow it to work with LockDown Browser. 

Accessibility limitations

According to Student Disability Services, Respondus may be extremely resource intensive and may cause the user’s computer to crash if the student has limited bandwidth or is using an older computer (this applies to all students, not just students with disabilities). Additionally, just like with any other classroom practice and technology, students with disabilities may need additional accommodations related to his approach and/or technology. Faculty should be aware that the use of Respondus may present a gray area where an alternative exam solution needs to be worked out between the instructor and the student. Faculty may reach out to the Student Disability Services to consult. 

Resources

• CTL approach to student privacy at Baruch: See the “Student privacy” section on COVID-19 Teaching Resources Page
• CTL guidelines about online exams and links to Blackboard tutorials on Teach Hybrid
• Creating open-ended questions (part 1 and part 2 from Prof. Will Millhiser)
• Exam Options, April 1, 2020, UC Davis

Other Related Readings

• • Online Test-Takers Feel Anti-Cheating Software’s Uneasy Glare, April 5, 2015, The New York Times.
  • Online Exam Proctoring Catches Cheaters, Raises Concerns, May 10, 2017, Inside Highered. 
  • University of California Santa Barbara Faculty Association Issues Letter Against the Use of ProctorU Testing Services, March 16, 2020, Daily Nexus
  • Privacy and the Online Pivot, March 25, 2020, Inside Higher Ed
  • Mass school closures in the wake of the coronavirus are driving a new wave of student surveillance, April 1, 2020, The Washington Post.
  • Our Bodies Encoded: Algorithmic Test Proctoring in Higher Education, April 2, 2020, Hybrid Pedagogy 
  • College Made Them Feel Equal. The Virus Exposed How Unequal Their Lives Are, April 4, 2020, The New York Times
  • Colleges Flock to Online Proctors, But Equity Concerns Remain, April 7, 2020, Education DiveThis option works especially well for small discussion-based classes, though it’s also effective for large lectures, especially if you have a moderator.