This article originally appeared in The Bar Examiner print edition, Winter 2022–2023 (Vol. 91, No. 4), pp. 28–31.
By Marilyn Wellington
Pilot Testing, Field Testing, and Prototype Testing: A Look at the Interconnected Research Phases for the New Bar Exam
As NCBE marches toward the July 2026 launch of the next generation of the bar exam, it has entered a new phase in the exam’s development: pilot testing. NextGen updates, programs, and webcasts have mentioned pilot testing, as well as field and prototype testing; each is a critically important phase in the new exam’s development. In this column I’ll dive deeper into these interconnected research testing phases to be performed over the next few years.1 Each round of research will provide qualitative and quantitative data and information to help ensure the next generation of the bar exam tests what it’s meant to and does so in a way that is fair and accessible to all examinees. As part of these testing phases, participants are asked to respond to survey questions developed by the NextGen research team to provide insight into participants’ thought processes as they respond to the draft exam questions. Survey responses will shape future item development and provide data and insights to ensure the fairness and validity of the NextGen exam.
Pilot testing is well underway, with two waves of pilot tests already done and continuing into 2023. This phase is intended to evaluate the form, organization, and structure of the exam items. Pilot testing participants are recent graduates and law students who are near to graduation, drawn from law schools across the country. Pilot testers have approximately two hours to answer a series of draft exam items and provide feedback.
Pilot testing allows us to gather data about how the new question formats perform in a real-world testing situation, which provides valuable information to help us finalize crucial details regarding what the NextGen exam will look like, including:
- The types of questions that will be used on the new exam. The NextGen exam will include new item types alongside those used on the current exam, such as essay and multiple-choice questions. New item types may include, for example, short answer questions or multiple-choice questions in a format that allows for more than one correct response. Pilot testing is an opportunity to collect both qualitative and quantitative data on how these new question types will perform on the exam and adjust them as necessary based on item performance and participant feedback.
- The best way to combine and sequence different types of questions. The NextGen bar exam will be an integrated exam, using mixed item sets rather than standalone questions. An integrated format allows us to provide item sets that more closely reflect situations that a newly licensed lawyer may see in practice. Whereas each of NCBE’s current bar exam components contains a single type of question (multiple choice, essay, or performance test), NextGen examinees will encounter a mixture of several different question types. For example, examinees might be presented with a scenario followed by a handful of short-answer questions, a prompt to draft a memo, and several multiple-choice questions. Pilot testing allows us to determine the best order in which to present those questions: for example, should an item set begin with constructed responses, move to short-answer questions, and finish with multiple choice? Or is another order better?
- The best way to provide examinees with supplemental materials. Another major change underway for the new exam is the plan to provide examinees with supplemental materials where appropriate, such as relevant portions of the Federal Rules of Evidence, so that they need to rely less on specific recall of legal doctrine details. During pilot testing, we can study the impact of providing such materials and determine how best to provide them in a way that is fair and equally accessible to all.
- How long we should expect it to take for examinees to complete new types of questions and question sets. Since we’ll be introducing types of questions that haven’t previously been used on the bar exam, and because different question types will be integrated, a key factor in pilot testing is determining the amount of time examinees need to respond to the exam’s new item sets in a test setting and what factors may impact the time required. For example, the provision of supplemental materials and ease in accessing them may influence the time needed. Ultimately, this information will help determine how much time should be allotted for the entire exam.2
As we move to field and prototype testing in 2024, we will have answered several crucial questions thanks to pilot testing, such as the format of item types, the best order in which to present item sets, how to provide supplemental materials in a way that is fair and accessible, and how much extra time the use of those materials might add to the total time it takes examinees to complete a set of questions. We’ll know, overall, how much time it takes examinees to complete the new types of test items. In addition, we will have begun assessment of the specific content of the questions themselves: the foundational skills and knowledge areas they assess.3
Field and Prototype Testing
Following a thorough analysis of the data received through pilot testing, the NCBE NextGen research team will have sufficient data to build multiple simulated exams for field and prototype testing. These phases complement each other, providing researchers with the qualitative and quantitative data needed to solidify the structure of the final NextGen exam.
Field and prototype testing involve larger numbers of examinees than pilot testing and are designed to help us know whether the specific questions or item sets that NCBE exam drafters prepare will assess examinees’ knowledge and skills level as intended. NCBE works with subject matter experts and provides them with training, guidance, and support to draft items for use on exam items.4 Before any item is included on a field or prototype test, our expert drafters and reviewers will have ensured the questions cover the topics within the Content Scope Outlines. What we will confirm at this point is that the questions are fair and within the appropriate level of difficulty to establish readiness to enter the practice of law. Field testing will consider:
- Whether the specific test items have the right level of difficulty to assess the requisite knowledge and skills of a competent newly licensed lawyer. The bar exam is a licensure exam. Its purpose is to provide jurisdiction admissions authorities with a tool to meet their responsibility to the public by assessing whether candidates for bar admission have the competency required to perform the activities typically required of a newly licensed lawyer. Given this, each test item needs to contribute to a total exam score that allows those authorities to determine which examinees have the knowledge and skills needed. For the exam to serve that purpose, each question should be one that qualified examinees can respond to correctly. If a question is too easy, such that any examinee can answer it correctly, it won’t provide information about who is minimally competent to practice law. The same is true of a question so difficult that no one can answer it. Analysis of participant performance on field testing will allow NCBE to determine which questions fall into each of these camps, and which are just right (the “Goldilocks test”), to make the necessary adjustments so that each test item will contribute meaningfully to a total exam score.
- Whether test items are shown to be fair to all examinees. Although test items should distinguish between those who are ready to enter the practice of law and those who aren’t, they should not make other kinds of distinctions, such as between examinees of different races, ethnicities, socioeconomic backgrounds, genders, or disability status. Field testing will provide data that allows us to ensure that an examinee’s ability to respond correctly to test items doesn’t change based on these irrelevant factors.
- NextGen exam scoring systems and test delivery. NextGen test item drafters and reviewers are developing scoring rubrics for each test item in concert with exam graders retained by jurisdictions. In field testing, NCBE will work with jurisdiction graders to give them the opportunity to grade the new integrated test items. Field testing will also allow us to try out a new exam delivery model. Although the exam will continue to be administered at jurisdiction-managed exam sites, the delivery method will no longer be paper-based. The NextGen exam will be a fully computer-based exam—computer delivery will be from a single platform that NCBE will provide. As part of field testing, NCBE will work with jurisdiction admissions authorities to administer the exam under this new model.
In addition, we expect that field and prototype testing will give jurisdictions and law schools the opportunity to see a full-length exam administered as it will be beginning in July 2026.5 The field and prototype exams will look and feel exactly like real exams and will help complete the final step of bringing a new bar exam into a reality: standard setting to help jurisdictions determine their new passing scores.
As a reminder, though NCBE develops the bar exam, it doesn’t determine what constitutes a passing score on the exam. Instead, each jurisdiction sets its own criteria for what constitutes a passing performance. On the current Uniform Bar Exam, jurisdiction-set passing scores range from 260 to 280. Jurisdictions consider a range of different issues and information when setting passing scores, but at the heart of this process is what’s known as a standard-setting exercise. For the NextGen exam, participant performance on the prototype test will be used to guide the standard-setting process for jurisdictions.6
The research testing described here plays a vital role in transitioning from plan to reality for the next generation of the bar exam. Each round of testing will provide the data, information, and insight to ensure that the new exam provides a fair and meaningful assessment to support jurisdictions in determining whether their examinees are ready to begin practicing law.
- For a high-level overview of the test development timeline, visit https://nextgenbarexam.ncbex.org/about/implementation-timeline/. (Go back)
- In the world of standardized testing, there is a distinction between speeded and nonspeeded tests. A good example of a speeded test is a typing test: the examinee’s speed is tested along with the accuracy of their performance. This is not the case for the bar exam, where the intent is for examinees to exhibit their ability to work at a speed that is reasonable for legal work while fully demonstrating their skills and knowledge. For a more in-depth discussion of speededness and the bar exam, see Mark A. Albanese, PhD, “The Testing Column: Speed (Not the Drug, and It Does Not Kill, but It Can Cause Stress),” 84(4) The Bar Examiner 7–41 (December 2015). (Go back)
- NCBE published draft NextGen Content Scope Outlines for public comment in spring 2022 (see https://nextgenbarexam.ncbex.org/csopc-register/). More than 400 comments were submitted. Final Content Scope Outlines—edited to incorporate public comment, detailed legal resource information, and other edits by subject matter experts—will be published in the second quarter of 2023. (Go back)
- For more information about the role of pretesting for the current exam, see C. Beth Hill, “MBE Test Development: How Questions are Written, Reviewed, and Selected for Test Administrations,” 84(3) The Bar Examiner 23–28 (September 2015); and Judith A. Gundersen, “MEE and MPT Test Development: A Walk-Through from First Draft to Administration,” 84(2) The Bar Examiner 29–34 (June 2015). (Go back)
- Jurisdictions will have the opportunity to begin administering the NextGen exam starting in July 2026, but NCBE anticipates that not all jurisdictions will participate in this first administration. (Go back)
- For more information on standard setting, see Michael T. Kane, PhD, and Joanne Kane, PhD, “Standard Setting 101: Background and Basics for the Bar Admissions Community,” 87(3) The Bar Examiner 9–17 (Fall 2018). (Go back)
To stay up to date on development of the future bar exam, subscribe to the NextGen website at nextgenbarexam.ncbex.org/subscribe.
Marilyn J. Wellington is the Chief Strategy and Operations Officer for the National Conference of Bar Examiners.
The Next Generation of the Bar Exam
In January 2021, the NCBE Board of Trustees approved the recommendations of NCBE’s Testing Task Force for the redesign of the bar examination to ensure that it continues to test the knowledge, skills, and abilities required for competent entry-level legal practice in a changing profession.
The board appointed an Implementation Steering Committee (ISC), which is charged with general oversight of the implementation of the findings and recommendations from the Testing Task Force study. NCBE has established multiple workgroups, working in consultation with the ISC, to develop the next generation of the bar examination and ensure a smooth transition for candidates, jurisdictions, and law schools. The workgroups include focus areas such as: test development and psychometrics; test delivery; diversity, fairness, and inclusion; and outreach. Workgroups focused on developing the content of the new exam and drafting exam questions are comprised of law professors, legal practitioners, and judges and justices.
Contact us to request a pdf file of the original article as it appeared in the print edition.