Test and Information Security: Centralizing Security Initiatives at NCBE

This article originally appeared in The Bar Examiner print edition, Spring 2018 (Vol. 87, No. 1), pp 46–49.

By C. Beth Hill

At NCBE, the security of our test materials, the prevention and detection of cheating, and the safeguarding of examinee and character report applicant data have always been of paramount importance, and responsibility for security has been widely distributed across the organization. In order to give exclusive focus to the ever-increasing threats to security, NCBE has recently decided to create the new position of Director of Test and Information Security, under which responsibility for all security-related activities will be centralized.

As I transition from my former role as the Program Director for the Multistate Bar Examination into my new role as NCBE’s Director of Test and Information Security, I will be evaluating NCBE’s current security efforts and assessing how NCBE can strengthen its security protocols and training to further minimize the security risks that threaten to undermine the integrity of the bar admissions process. In this article I discuss some of these current security efforts and upcoming initiatives.

NCBE Security Resources for Jurisdictions

NCBE already offers a variety of resources to guide exam security efforts in the jurisdictions. We provide an annually updated Supervisor’s Manual for those supervising the administration of the bar exam, which details the protocols for a standardized, secure exam administration across both the essay and the MBE portions of the bar exam. The Supervisor’s Manual details the supervising personnel and proctor duties from pre- to post-administration of the exam. Strict adherence to the required protocols is essential to the secure administration of the exam, as test supervisors carry the major responsibilities of ensuring that no examinee has prior access to the tests, protecting the security of test materials throughout the administration, providing a testing environment that is distraction-free, ensuring the efficient and correct administration of each test, and ensuring that all MBE test materials are returned promptly after the administration.

For example, the Supervisor’s Manual outlines the protocol for assigning seats in advance and for checking in examinees with proper identification on testing days. In addition, it provides a detailed list of items that examinees are prohibited from bringing into the testing room (e.g., no cell phones, fitness trackers, or watches of any kind are allowed). This list is also posted on NCBE’s public website, so examinees are able to review the list at any time. And the manual details the procedures for collecting test booklets and returning MBE booklets—all of which are individually scanned by NCBE’s secure printer when they are returned to verify that each and every one was collected—and for returning MBE answer sheets.

The Supervisor’s Manual is to be used in conjunction with other resources and educational opportunities that NCBE provides. For example, we have a series of training videos designed to guide supervising personnel through the many steps involved in exam administration. NCBE has other resources on our secure website to assist test supervisors, including key exam-related dates and deadlines, the Exam Orders Site, and post-administration forms. NCBE also hosts mini-seminars for new bar admission administrators to provide in-person training on significant issues pertaining to bar admissions, including test security.

The resources NCBE provides for jurisdictions are essential, because once NCBE’s examinations are printed, the administration of the exams is handled by others—the jurisdictions’ bar admission administrators, staff, proctors, and board members. This means that exam security is a partnership.

As part of our enhanced security initiatives, we are looking at providing more security-related training materials and videos to assist test supervisors in ensuring that proctors and staff are well versed in securely handling test materials and deterring and detecting cheating, as well as “item harvesting,” the attempt by examinees to commit questions to memory, or capture them electronically or by other means, for dissemination to future examinees.

Post-Exam Cheating Analysis

NCBE’s exam security partnership with jurisdictions continues after the administration of the exam has occurred. After each bar exam, each jurisdiction completes an NCBE Irregularity Report to document whether or not any incidents occurred, including, but not limited to, suspected cheating, possession of a prohibited electronic device, or possible theft of test materials. If an examinee is suspected of copying from another examinee on the MBE, a jurisdiction can request a detailed answer sheet comparison analysis. This analysis, performed by Mark Albanese, Ph.D., NCBE’s Director of Testing and Research, provides a statistical comparison between the answer choices of the suspected copier and the potential source.1

In my new role, I will be investigating ways to continue to improve the communication about exam-day incidents as well as how to deter such incidents in the first place. While there are still the “old-­fashioned” cheating methods, such as copying from a neighbor’s answer sheet or bringing in notes on a scrap of paper, the testing industry is increasingly challenged by new technologies that can facilitate cheating. For example, a simple Google search yields a number of products specifically designed to enable cheating, such as spy glasses for capturing exam questions and earpieces for receiving assistance via electronic communication.

Monitoring Exposure to Exam Questions

Before and after each bar exam, NCBE contracts with a web crawling service to help identify instances where examinees go beyond discussing the bar exam in general terms and delve into re-creating exam questions from memory or from a surreptitious capture. Any disclosure, whether done innocently or knowingly, is in violation of NCBE’s copyright and a threat to the future viability of the multiple-choice questions.

When our copyrighted material is disclosed online, we issue takedown notices to the website administrator. In some instances, when we have been able to identify the person who posted the infringing material, we have notified the jurisdiction in which the individual has applied for admission about a possible violation of the jurisdiction’s character and fitness requirements.

One area in which NCBE would like to redouble its efforts is in the “frequent flyer” zone. We use this term to refer to individuals who take the bar exam multiple times, often in multiple jurisdictions, and who may be doing so to capture questions for dissemination to future examinees instead of making a real attempt to answer the questions correctly and pass the exam. Investigating examinees who may not be bona fide candidates for admission takes a collaborative approach between the jurisdictions and NCBE. Occasionally, I am contacted by jurisdiction bar admission administrators who suspect, based on information reported by the examinee or as a result of the jurisdiction’s character and fitness investigation, that they may have an examinee who is not a bona fide candidate. Not infrequently, the examinees have previously tested multiple times in one or more jurisdictions. Consistent use of the NCBE Number (a number that serves as a unique identifier for bar admission candidates) helps considerably in our effort to begin to combine information across jurisdictions and administrations to identify such examinees.

In addition, NCBE hosts a Cross-Reference Service on its secure website, which jurisdictions may use to record where individuals have submitted applications for admission. This service can provide useful information about individuals who have applied for admission in multiple jurisdictions; however, not all jurisdictions are using the service to share their data. Consistent and regular use of the system by all jurisdictions would be helpful in making the Cross-Reference Service a resource for identifying potential non–bona fide candidates and for providing jurisdictions with information they could use to investigate an applicant’s bona fide status before the applicant is allowed to sit for the exam and gain access to test materials.

Conclusion

NCBE values its partnership with jurisdictions to ensure the secure administration of the bar exam and the prevention and detection of cheating, thereby maintaining the integrity of the bar admissions process. As NCBE’s new Director of Test and Information Security, I welcome your ideas on strengthening our partnership to combat threats to the security of the bar exam.

Contact us to request a pdf file of the original article as it appeared in the print edition.