Testing during the COVID-19 Pandemic: A Survey of Six Testing Organizations

This article originally appeared in The Bar Examiner print edition, Winter 2021-2022 (Vol. 90, No. 4), pp. 16–22.By Rachel R. Watkins Schoenig

On January 31, 2020, the US Secretary of Health and Human Services declared a public health emergency due to the virus that would eventually become known as COVID-19.1 Less than two months later, states began issuing lockdown orders to help contain transmission of the disease.2 As days turned into weeks and the virus spread unchecked around the globe, one thing became clear: the world wasn’t returning to normal anytime soon. With schools, test centers, and other large venues remaining closed, certification and licensure organizations grappled with whether and how to offer exams amid concerns about the health and safety of test takers, proctors, and employees.

This article explores the responses to pandemic restrictions undertaken by six testing programs—four in the United States, one in Canada, and one in the United Kingdom. The responses undertaken by these six organizations—which span medicine, engineering and surveying, auditing, and law school admission—represent unprecedented efforts to make assessments available to candidates at a time when people around the world were struggling with the changes the pandemic wrought.

Each section begins with brief information about the testing organization, followed by a summary of the responses undertaken, including statements from a representative of the organization from which the information was obtained. A summary of NCBE’s response to the challenges posed by the pandemic for the bar exam is in the sidebar.

NBME, US

NBME (the National Board of Medical Examiners) offers a selection of assessments and educational services for students, professionals, educators, regulators, and institutions dedicated to the evolving needs of medical education and health care. It offers a broad suite of exams, including the United States Medical Licensing Examination^® (USMLE^®), which is offered in three “steps” at computer-based testing centers, and NBME^® Subject Exams, most of which are proctored exams offered in medical schools.

Pandemic-related restrictions and social distancing requirements caused significant upset across the medical education and licensing communities. Like most testing programs, NBME found its test delivery operations completely upended. NBME, however, faced additional pressure. Because medical school students, medical residents, and international medical graduates are considered essential workers, the pressure to provide some means of measuring competency during the pandemic was significant. As the virus quickly spread across the globe and hospitals strained to serve patients, NBME rapidly re-evaluated and customized its pandemic response plans. “Our priority was to help a disrupted medical education system in ways where we could make a difference,” explains Mike Jodoin, Senior Vice President for NBME Customer and Portfolio Management.

The USMLE is an exam used to evaluate applicants’ competence for medical licensure, and scores are reported to licensure boards in the United States and its territories. USMLE exams are typically delivered by appointment through third-party computer-based test centers. When test centers initially shut down, it was anticipated they would be closed for two months. As COVID-19 cases continued to rise, however, and test centers remained closed or capacity was severely limited, USMLE exams were frequently canceled and rescheduled.

Rather than continue to wait for test centers to reopen, NBME rapidly explored alternative delivery options. It decided remote ­proctoring­­­—which allows individuals to take an exam on a computer at their home, office, or other quiet location outside of a test center—was not a good fit for the USMLE at that time. Instead, in partnership with the medical education community, NBME decided to offer the USMLE on specific test event days for groups of test takers at regional medical schools with in-person proctoring.

NBME closely collaborated with its third-party delivery vendor and medical educators to forecast needs and allocate seats. The first of six pop-up exam sites at medical schools across the country was ready for testing within two weeks of reaching an agreement. NBME also coordinated four exam administration events during July and August. “As a brand-new type of administration for the USMLE, event testing required significant changes, and leveraging technology in place at the medical schools,” recalls Jodoin.

In addition to delivery changes for the USMLE, NBME also provided alternative delivery methods for its medical school exams. NBME Subject Exams are typically given during in-person proctored sessions at medical schools, often as an end-of-course assessment. Initially, NBME postponed delivery of Subject Exams due to the pandemic. Soon, it became clear that medical schools would need to administer Subject Exams before the end of the semester.  NBME began assembling teams to explore remote proctored solutions to help medical school educators continue to advance learners toward caring for patients.

To accommodate those needs, NBME Subject Exams were delivered remotely to students through web-conferencing with remote proctoring, rather than in the classroom setting. Within four weeks, 27 Subject Exams were available to be administered to students remotely.

In addition to new testing locations and modes of delivery, NBME also provided free NBME Self-Assessments to help students practice and assess their readiness for their upcoming USMLE examinations. This offered some relief to those who may have had exams canceled or postponed due to test center closures. During this period, students ordered over 400,000 free self-assessment forms.

“NBME was focused on providing solutions that best served the medical community as a whole based on the challenges faced,” Jodoin says. Those challenges gave NBME opportunities to partner with medical educators in new ways and collaborate with all stakeholders to triage needs under very difficult circumstances. Importantly for the health community, NBME maintained exam integrity and validity during this process. Several analyses showed no meaningful performance differences when comparing performance during the pandemic to pre-pandemic administrations.

Schools and students both responded positively to NBME’s timely efforts. And according to Jodoin, the organization is continuing to learn from the changes required during the pandemic. “COVID-19 accelerated our vision for innovations already identified. Everyone has learned a lot as individuals and as professionals. As the organization continues to move forward, our goal will be to use that experience and continue innovating the science of professional assessment in new and bold ways.”

The Medical Council of Canada

The Medical Council of Canada (MCC) assesses medical students and graduates through its examinations, granting the Licentiate of the Medical Council of Canada (LMCC) to physicians who have met its requirements. Prior to the pandemic, physicians had to pass two qualifying exams to apply for the LMCC: the Medical Council of Canada Qualifying Examination (MCCQE) Parts I and II, and the National Assessment Collaboration (NAC) exam.

MCCQE Part I is typically a one-day computer-based exam that assesses the critical medical knowledge of a medical student completing their medical degree in Canada. Under normal circumstances, MCCQE Part I exams are available up to five times a year within 2- to 6-week testing windows. Testing occurs at third-party test centers in Canada and 80 other countries.

As the pandemic forced test center closures, the MCC initially decided to pause the MCCQE Part I for six weeks. As time went on and the need for health professionals continued to rise in the face of the pandemic, the MCC sought alternatives to test center delivery as a way to timely and safely move candidates along the route to licensure.

The MCC chose to implement remote testing with online proctoring as an alternative delivery option. For high-stakes exams, remote testing using online proctoring helps ensure the security and integrity of the test. MCC also addressed pent-up demand caused by test center closures by adjusting and expanding the exam window by several weeks. The MCC communicated extensively with candidates regarding the changes and its goal to safely test as many Canadian medical school graduates as possible prior to the start of residency training.

The MCC maintained identical test content specifications and test design across both in-person computer-based and remote delivery testing options. It also used the same exam interface and scoring procedures. The MCC analyzed significant amounts of data for both test delivery options to provide evidence of comparability and score validity. Although test takers who completed the exam remotely did have higher levels of lag times and test driver disconnections (either due to computer power issues or internet disruptions), these did not appreciably impact key psychometric properties for the exam. The MCC concluded there were no appreciable differences in pass rates and average total scores between in-person and remote delivery.

The MCC plans to continue to offer both in-person and online testing opportunities. “Our experience suggests that offering a high-stakes examination remotely is a viable alternative to supplement the traditional test-center based approach. The convenience of ‘anytime/anywhere’ testing provides test taker flexibility from a number of perspectives, including choice of location and convenience of start time,” states Dr. Maxim Morin, Manager, Psychometrics and Assessment Services at the MCC. “It also eliminates travels costs and leads to a potentially more relaxing experience for test takers than writing in a testing center.”

The General Medical Council, UK

The work of the United Kingdom’s General Medical Council (GMC) includes registering doctors, regulating medical education, and setting standards. Prior to the pandemic, candidates were required to pass the Part 1 exam, which is a paper-based exam delivered four times a year, and the Part 2 exam, which requires candidates to engage in a clinical-like evaluation of practical skills using standardized patients as part of the exam process. The Part 2 exam is delivered hundreds of times each year at two test centers in central Manchester, England.

Given its critical role in the health community, GMC was committed to ensuring testing could continue during the pandemic. The Part 1 exam was offered with social distancing and other protections in centers where large gatherings were still allowed. For two administrations, the number of test items was reduced to allow for splitting the candidates into a morning and afternoon cohort to further facilitate social distancing. 

The Part 2 exam, which requires close physical contact, was suspended from March to August 2020. This provided time for GMC to reimagine the Part 2 exam experience and account for social distancing requirements. GMC remodeled one exam center, which opened in the fall of 2020, and designed an entirely new center, which opened in June 2021. Rooms are larger for the practical skills stations, and the exam now includes telephone and video consultations, which have become more common in medical practice during the pandemic. Raters also score candidates remotely, further minimizing the level of personal contact.

GMC now operates two COVID-secure centers in adjacent buildings which, combined, gives it the same capacity as a single center did pre-pandemic. The new centers enable GMC to continue to assess competency and enable doctors to gain registration while adhering to any COVID-19 regulations. GMC phased out social distancing completely in November 2021 and now relies upon the use of rapid COVID-19 tests to ensure safety. It has three dedicated testing rooms, and test takers are provided the COVID-19 test on arrival. With the continued use of the temporary clinical assessment center, its new capacity is 50 percent greater than before the pandemic.

GMC employed the same standard-setting approaches it used pre-pandemic and incorporated a standard error of measurement into the passing score. “This gave us security that those who passed met the required standard on all occasions,” explains Richard Hankins, head of Assessment for GMC. While GMC did see an increase in the pass rate for the Part 1 exam, it was still within the historical margins for the exam and likely explained by the additional time candidates had to prepare due to delays caused by test center closures and limits on social activities.

GMC’s relatively low-technology response to the pandemic enabled it to successfully begin testing quickly and retain all the key elements of the exam program. Hankins explains the sense of urgency driving the exam changes. “We had some candidates who were stranded in the UK and others who had job offers in place. It was important to us to resume testing as soon as possible, both for the individual candidates and for the health community.”

The National Council of Examiners for Engineering and Surveying, US

The National Council of Examiners for Engineering and Surveying (NCEES) develops and administers the exams used for engineering and surveying licensure in the United States and its territories. For the last several years, NCEES has been in the process of converting its paper-and-pencil exams to computer-based delivery. Today, the majority of its exams are offered via computer in test centers, but some large exams are still offered on paper. NCEES’s largest exams are offered year-round, and smaller exams are offered during single-day testing windows. Exams vary from six to nine hours in length, with a break occurring roughly halfway through the exam.

Because NCEES offers both computer-based and paper-and-pencil exams, it juggled a variety of responses to the pandemic. It initially suspended computer-based exam delivery during the beginning of the pandemic when its third-party delivery centers were closed. NCEES opted to wait until computer-based test center delivery resumed rather than to offer exams remotely.  “In discussions with our testing vendor, it was determined that our exams are currently too long to be efficiently administered with remote proctoring,” explains Bob Whorton, Compliance and Security Manager at NCEES. In addition, NCEES was concerned that remote delivery could impact security and exam content.

Paper-and-pencil exams were similarly canceled in the spring of 2020. NCEES later made the decision to host its larger paper-and-pencil exams in October under newly established COVID-19 guidelines, including capacity limitations, required face masks, and social distancing, where permitted. NCEES worked closely with chief proctors to review the new guidelines. In states where large gatherings were prohibited, COVID-compliant exam sites were added in adjacent states. NCEES also added a January 2021 paper-and-pencil administration to help increase testing opportunities available to candidates.

“Overall, candidates had a great deal of nervousness in sitting for the exams,” explains Whorton. “Many candidates were just not ready to sit in a large room with a lot of other people for an entire day.” Those health concerns factored into the decisions the organization reached in when and how to resume testing.

NCEES also modified aspects of its exam development process, moving all exam development meetings with subject matter experts from face-to-face to virtual. It also has one exam that contains constructed response questions that are manually graded during a face-to-face meeting twice per year, for which it developed an online grading system so that the papers could be graded virtually.

According to Whorton, the adjustments made by NCEES were well-received by both candidates and jurisdictions. “Test takers for the most part appreciated our efforts and were just happy to be able to test, even if at a later date and under different conditions. Our member licensing boards understood and appreciated the fact that we were able to offer the tests at all.”

Overall, licensing boards did not waive licensure exam requirements, even during the height of the pandemic restrictions. Some jurisdictions, however, did modify their processes and issue licenses to individuals applying for admission by comity without requiring applicants to complete jurisdiction-specific exams, giving them a specific period of time to complete the exams after issuance of the license instead of requiring completion before obtaining the license.

NCEES plans to continue transitioning its paper-and-pencil exams to computer-based delivery modes. Whorton notes, “One key learning from the pandemic was the extreme difficulty involved in implementing a large-scale backup plan for paper-and-pencil exams offered in large group settings across the country. We are fortunate that we were nearing the completion of our multiyear effort to transition our exams to a computer-based format when the pandemic started. We were able to fast-track the process, which will allow us to fully complete the transition ahead of schedule.”

The Institute of Internal Auditors, US

The Institute of Internal Auditors (“The IIA”) provides standards and guidance to the internal audit profession through affiliated internal audit institutes around the world. It awards the Certified Internal Auditor credential as a means of demonstrating internal audit knowledge, skills, and competencies, and offers the Certified Internal Auditor exam in 14 different languages at authorized vendor sites around the world.

At the onset of the pandemic, The IIA initially suspended testing due to test center closures. By mid-March 2020, however, the organization began to explore moving to online proctored testing to address the global demand for candidate testing.

The IIA reviewed a number of options of online proctored systems with a focus on the security of test delivery. To adapt to online testing, it moved from desktop delivery to a driver that supported cloud delivery, published more than 100 new test forms, and modified some testing rules. The IIA was able to launch online testing starting on April 27, 2020. The launch enabled candidates to choose to either test in an open test center or test online in a quiet location at a time of their choosing.

“This change enabled us to truly continue to accommodate our test takers seeking certification 24/7/365,” explains Nicole Rhyne, Director of Exam Development and Security at The IIA. “In the past, a test center may have closed at 5 pm in a candidate’s geographic area, but candidates can now test at 10 pm if they choose to do so.”

Initially, The IIA thought the online proctored option would be temporary and would conclude in December 2020; however, it has continued to offer this online option to date. “We have evaluated data on a regular basis to decide whether to stay online. We looked at what the test takers are selecting and how the test was performing. We tracked pass rates across exams, languages, and regions, and conducted forensic analysis on the results to verify that pre-pandemic and post-pandemic rates were within the expected statistical ranges. There have been no changes to pass rates and performance on the exam.”

Regarding the work that went into pivoting the Certified Internal Auditor exam operations in response to the pandemic, Rhyne says, “It was absolutely the right thing to do to make that transition and do that work and get tests up and running as quickly as possible so candidates could get back on the path as quickly and seamlessly as possible. Transitioning to this process was at times bumpy and difficult, but we were doing the right thing for the candidates.”

The organization has received positive feedback from candidates and has learned a significant amount that can be leveraged for the future. Rhyne states, “The past 18 months will impact strategic planning for The IIA. It’s also positioned the organization to more quickly and holistically evaluate and adapt to changes.” 

The Law School Admission Council, US

In contrast to the high-stakes licensure exam programs discussed above, the Law School Admission Test (LSAT) migrated to a shortened version of its exam for remote delivery modality.

The Law School Admission Council (LSAC) provides services for prospective law students, including administering the LSAT. Prior to the pandemic, the LSAT was offered eight or nine times per year in North America, and four times per year at international sites. The exam was delivered in person at large test event centers using tablets loaded with proprietary testing materials.

When public health authorities prohibited large public gatherings due to COVID-19, LSAC was forced to cancel its March, April, and June 2020 in-person LSAT administrations, leaving 40,000 applicants unable to test and apply to law school. “At the time, we were concerned it would have a devastating impact on the 2020 admission cycle,” explains Susan Krinsky, Executive Vice President for Operations and Chief of Staff at LSAC.

To address the needs of test takers, LSAC developed a new online admissions test, which was launched the week of May 18, 2020. “Our team worked around the clock for two months to design, develop, and test the online LSAT-Flex,” Krinsky states. Like many testing organizations, LSAC continued to hope it would be able to return quickly to some form of in-person, socially distanced testing. Over time, however, it became clear the pandemic was not receding, and LSAC was forced to cancel all in-person exam events and replace them with the online LSAT-Flex.

LSAC chose to use live online proctoring to monitor the LSAT-Flex exam administration. The previous in-person LSAT was designed for approximately three hours with a break. LSAC designed the LSAT-Flex exam so it could be completed in two hours without a break. Both exams, however, tested the same skills, and both used questions that had been extensively tested and analyzed through the same rigorous item-development process.

LSAC has seen a slight increase in mean scores for online testing. Krinsky notes this may be due to test takers feeling more comfortable taking the test in their own home rather than traveling to a test center and its associated hassles, and testing in a large, crowded, unfamiliar space. It may also be because the lockdown experienced during the pandemic provided candidates with more time to prepare for the exam. Based on survey data, candidates spent 25–30% more time in preparation than prior to the pandemic.

Krinsky states that most test takers were grateful for the opportunity to test in their own homes; in surveys, a majority of test takers said they preferred online testing to in-person testing.  LSAC plans to continue with online testing at least through June 2023.

Krinsky says the entire legal education ecosystem continues to work together to confront the challenges posed by the pandemic. “Schools have been very appreciative of how quickly we were able to design and deploy the online test, particularly given the devastating potential impact COVID-19 could have had on the fall 2020 admission cycle,” Krinsky states. “The past two years have been tragic and exhausting, but the entire legal education community has rallied together to support applicants and ensure that their law school dreams can continue to move forward.”

Conclusion

The COVID-19 pandemic has had a major impact on many aspects of education and assessment, including the administration of licensure, credentialing, and admissions exams. For testing programs, creating access to exams and adjusting to ever-changing regulations and health concerns required tremendous effort by testing professionals. Across the industry, testing programs and vendors rose to the challenge and reimagined traditional test design, delivery, and proctoring options to provide access to important high-stakes exams. Many of the changes implemented will have a lasting impact on the testing industry, providing new opportunities to serve test takers and enhancing access to assessment, including access for those in underrepresented populations.