This article originally appeared in The Bar Examiner print edition, Spring 2018 (Vol. 87, No. 1), pp 55–58.
By Mark A. Albanese, Ph.D.
Making and administering a bar examination is not for the fainthearted. There is a lot at stake, which requires that the utmost care be taken in the design, administration, and scoring of the examination. It is only through the coordinated and careful efforts of all those involved in the process that scores from the bar examination can provide confidence that candidates who achieve at passing levels are competent for practice. We may not be superheroes, but the bar admission administrator and NCBE are the dynamic duo that together can make this happen. In this article we hope to show how we provide synergy to each other in ensuring that candidates for the bar are examined in a fair and rigorous manner and that the passing decisions made by the jurisdictions provide protection for the welfare of the public.
NCBE’s Role and What We Provide for Jurisdictions
NCBE’s services to jurisdictions extend far beyond developing and producing high-quality test materials for jurisdictions to use in evaluating the proficiency of bar candidates. NCBE provides support to bar admission administrators before, during, and after each exam administration. This support includes the provision of detailed guidelines and required protocols for test administration and being at the ready on exam days to assist with any issues that may arise. Following the exam, NCBE provides training to jurisdiction graders for grading the written component of the exam. Our measurement staff members score the Multistate Bar Examination and scale the jurisdiction-graded written component of the exam to the MBE in order to produce scaled scores that jurisdictions use in determining whether examinees have passed the exam. We also conduct character and fitness investigations on behalf of jurisdictions and offer an online character report application and online access to character and fitness services for the jurisdictions’ bar candidates.
[J]urisdictions do not just receive high-quality test materials from NCBE; they receive access to a full-service organization that is dedicated to helping bar admission administrators do their jobs to the best of their abilities.
When something unusual occurs during a jurisdiction’s exam administration—which could be anything from a natural disaster to an examinee emergency to an exam facility problem (e.g., power outage, temperature control malfunction, water leak)—our research group is here to help administrators figure out what to do about providing scores in the wake of such a disruption. Over the past few years, the most common disruption has been noise from a variety of sources, such as nearby construction or another group using the exam facility simultaneously. There have also been cases of examinees having issues/emergencies such as urgent health situations or behavioral incidents. Generally, the question in such cases is whether the disruption affected the performance of examinees who were exposed to the disruption. Depending upon the location and duration of the disruption, we may perform comparisons between performance in the morning session of the exam with that in the afternoon session if the disruption only affected one or the other session. If the exam was administered in different rooms in the same facility or in different locations in the same vicinity, we can determine if those exposed to the disruption had a different pattern of performance than those who tested in a different room or location. How we investigate the effect of a disruption is very individualized and depends upon the particular situation. It isn’t unusual for us to run this type of analysis for a few jurisdictions after each exam administration.
If an administrator suspects that an examinee has cheated on the MBE by copying from another examinee, we perform cheating analyses that can reveal similar answer patterns between the suspected cheater and one or more neighboring examinees. We generally run such analyses at the request of two to seven jurisdictions after each bar examination.
We are also here to help jurisdictions make decisions about the future of their bar examinations. One of the most common requests that arise is to research the impact of a change to the weighting of the MBE and the written component of the exam in computing the scaled total score. In particular, jurisdictions contemplating adopting the Uniform Bar Examination (UBE) have been interested in knowing how their pass rates for different demographic groups of examinees would be affected if they changed their weighting to that required for generating a UBE score (i.e., MBE weighted 50%, the Multistate Essay Examination weighted 30%, and the Multistate Performance Test weighted 20%). We run formal research studies for jurisdictions on a regular basis, and we have seen a steady rise in such studies in recent years from about 12 in 2014 to nearly two dozen in 2017.
So, jurisdictions do not just receive high-quality test materials from NCBE; they receive access to a full-service organization that is dedicated to helping bar admission administrators do their jobs to the best of their abilities. And the bar administrator’s job is the focal point in the bar admissions process.
The Critical Role of the Bar Admission Administrator
The bar admission administrator’s role as it pertains to the mechanics of the bar examination (i.e., in addition to the political and budgetary aspects of the examination) is considerable. The administrator is responsible for arranging all logistics pertaining to the administration of the exam and for ensuring that exam administration takes place in a manner that ensures a level playing field for all exam takers. In addition, the administrator is responsible for managing the grading of the written component of the exam using consistent criteria and grading judgments; managing the data files that result; ensuring that the results from the different parts of the exam are combined according to the rules of the jurisdiction; applying the jurisdiction’s passing standard to the total score obtained from the exam to determine who passes and fails; and communicating the results to the examinees.
Each of these tasks is critical to the successful administration of a high-stakes examination and involves multiple steps. For instance, arranging the logistics of the exam administration involves such steps as reserving a testing facility—one with ample seating that meets minimum standards for providing workspace for each examinee and minimizing the ability to see another examinee’s work; creating a seating plan and attendance roster; formulating a test booklet distribution plan; providing rooms and extra time for examinees who have been granted disability accommodations; hiring and training proctors; inspecting all testing rooms for appropriate physical conditions (e.g., lighting, seating, visibility and functionality of clocks); securely managing the receipt, storage, distribution, and collection of test booklets and answer sheets; and accounting for all test materials after the examination and securely returning them to NCBE for processing.
And, in addition to the tasks identified here related to the mechanics of the bar examination, administrators manage and oversee the review of candidate applications to certify that each candidate meets the jurisdiction’s character requirements, is fit to practice law, and is otherwise eligible to obtain a license to practice law. Combine all these responsibilities, and it is clear that the bar administrator has a big job.
NCBE’s Support for Administrators
Recognizing the critical function of the bar administrator in this process, NCBE takes an active role in helping new administrators learn their jobs and in promoting best practices. To that end, NCBE regularly hosts a mini-seminar for new administrators to introduce them to significant issues in bar admissions and to help orient them to the myriad tasks that must be undertaken for a successful exam administration. We publish an annually updated Supervisor’s Manual for those supervising the administration of the exam detailing the protocols to be followed to ensure a standardized, secure exam administration. We sponsor a wide range of educational events addressing bar admission topics, as well as producing this publication. We also host an electronic mailing list for administrators, thereby providing them with the opportunity to share their experiences with and seek information from one another, and we maintain a secure website for them to access important information and services. We have staff members available in all departments to provide assistance and to answer questions as they arise.
The Dynamic Duo: Partners in Security
While NCBE does all it can to help prepare and assist bar administrators, it is up to the bar administrators to do the deed (or deeds). We view this partnership to be essential for the health of the bar admissions process and the legal profession. Due to the high-stakes nature of the bar examination, the secure handling of NCBE test materials and the prompt identification of test irregularities (e.g., suspected cheating, examinee emergencies, noise disruptions) are key components of the relationship that NCBE considers especially critical. (For more on the security partnership between NCBE and the jurisdictions, as well as the recent centralization of all security-related activities under NCBE’s new Director of Test and Information Security position, see C. Beth Hill’s article in this issue.) Failure to maintain high security has the potential to compromise, undermine, or even invalidate the bar examination. If any examinee has access to NCBE test materials in advance of the examination, he or she will have an unfair advantage. If during the examination, any examinee is able to obtain answers through some means beyond his or her own individual interaction with the test materials, the scores on the bar examination will be compromised.
NCBE relies on bar administrators to have vigilant proctors in place on exam day and to submit NCBE Irregularity Reports to document any test irregularities to enable us to ensure that the scores are meaningful. Irregularity Reports alert us not only to situations where cheating may have occurred but to any occurrence that could jeopardize the standardization of the testing process. Despite best efforts to ensure that the testing conditions are uniform across all jurisdictions, things can happen that are beyond all control. Irregularity Reports enable NCBE to adjust for these unexpected events.
It has been said that it takes a village to raise a child. Well, it takes bar admission administrators and NCBE to orchestrate a bar examination. Together we rock.
Contact us to request a pdf file of the original article as it appeared in the print edition.
