The Next Generation of the Bar Exam: Quarterly Update

This article originally appeared in The Bar Examiner print edition, Winter 2022–2023 (Vol. 91, No. 4), pp. 28–31.

By Marilyn Wellington

Pilot Testing, Field Testing, and Prototype Testing: A Look at the Interconnected Research Phases for the New Bar Exam

As NCBE marches toward the July 2026 launch of the next generation of the bar exam, it has entered a new phase in the exam’s development: pilot testing. NextGen updates, programs, and webcasts have mentioned pilot testing, as well as field and prototype testing; each is a critically important phase in the new exam’s development. In this column I’ll dive deeper into these interconnected research testing phases to be performed over the next few years.1 Each round of research will provide qualitative and quantitative data and information to help ensure the next generation of the bar exam tests what it’s meant to and does so in a way that is fair and accessible to all examinees. As part of these testing phases, participants are asked to respond to survey questions developed by the NextGen research team to provide insight into participants’ thought processes as they respond to the draft exam questions. Survey responses will shape future item development and provide data and insights to ensure the fairness and validity of the NextGen exam.

Pilot Testing

Pilot testing is well underway, with two waves of pilot tests already done and continuing into 2023. This phase is intended to evaluate the form, organization, and structure of the exam items. Pilot testing participants are recent graduates and law students who are near to graduation, drawn from law schools across the country. Pilot testers have approximately two hours to answer a series of draft exam items and provide feedback.

Pilot testing allows us to gather data about how the new question formats perform in a real-world testing situation, which provides valuable information to help us finalize crucial details regarding what the NextGen exam will look like, including:

1. The types of questions that will be used on the new exam. The NextGen exam will include new item types alongside those used on the current exam, such as essay and multiple-choice questions. New item types may include, for example, short answer questions or multiple-choice questions in a format that allows for more than one correct response. Pilot testing is an opportunity to collect both qualitative and quantitative data on how these new question types will perform on the exam and adjust them as necessary based on item performance and participant feedback.
2. The best way to combine and sequence different types of questions. The NextGen bar exam will be an integrated exam, using mixed item sets rather than standalone questions. An integrated format allows us to provide item sets that more closely reflect situations that a newly licensed lawyer may see in practice. Whereas each of NCBE’s current bar exam components contains a single type of question (multiple choice, essay, or performance test), NextGen examinees will encounter a mixture of several different question types. For example, examinees might be presented with a scenario followed by a handful of short-answer questions, a prompt to draft a memo, and several multiple-­choice questions. Pilot testing allows us to determine the best order in which to present those questions: for example, should an item set begin with constructed responses, move to short-answer questions, and finish with multiple choice? Or is another order better?
3. The best way to provide examinees with supplemental materials. Another major change underway for the new exam is the plan to provide examinees with supplemental materials where appropriate, such as relevant portions of the Federal Rules of Evidence, so that they need to rely less on specific recall of legal doctrine details. During pilot testing, we can study the impact of providing such materials and determine how best to provide them in a way that is fair and equally accessible to all.
4. How long we should expect it to take for examinees to complete new types of questions and question sets. Since we’ll be introducing types of questions that haven’t previously been used on the bar exam, and because different question types will be integrated, a key factor in pilot testing is determining the amount of time examinees need to respond to the exam’s new item sets in a test setting and what factors may impact the time required. For example, the provision of supplemental materials and ease in accessing them may influence the time needed. Ultimately, this information will help determine how much time should be allotted for the entire exam.2

As we move to field and prototype testing in 2024, we will have answered several crucial questions thanks to pilot testing, such as the format of item types, the best order in which to present item sets, how to provide supplemental materials in a way that is fair and accessible, and how much extra time the use of those materials might add to the total time it takes examinees to complete a set of questions. We’ll know, overall, how much time it takes examinees to complete the new types of test items. In addition, we will have begun assessment of the specific content of the questions themselves: the foundational skills and knowledge areas they assess.3

Field and Prototype Testing

Following a thorough analysis of the data received through pilot testing, the NCBE NextGen research team will have sufficient data to build multiple simulated exams for field and prototype testing. These phases complement each other, providing researchers with the qualitative and quantitative data needed to solidify the structure of the final NextGen exam.

Field and prototype testing involve larger numbers of examinees than pilot testing and are designed to help us know whether the specific questions or item sets that NCBE exam drafters prepare will assess examinees’ knowledge and skills level as intended. NCBE works with subject matter experts and provides them with training, guidance, and support to draft items for use on exam items.4 Before any item is included on a field or prototype test, our expert drafters and reviewers will have ensured the questions cover the topics within the Content Scope Outlines. What we will confirm at this point is that the questions are fair and within the appropriate level of difficulty to establish readiness to enter the practice of law. Field testing will consider:

1. Whether the specific test items have the right level of difficulty to assess the requisite knowledge and skills of a competent newly licensed lawyer. The bar exam is a licensure exam. Its purpose is to provide jurisdiction admissions authorities with a tool to meet their responsibility to the public by assessing whether candidates for bar admission have the competency required to perform the activities typically required of a newly licensed lawyer. Given this, each test item needs to contribute to a total exam score that allows those authorities to determine which examinees have the knowledge and skills needed. For the exam to serve that purpose, each question should be one that qualified examinees can respond to correctly. If a question is too easy, such that any examinee can answer it correctly, it won’t provide information about who is minimally competent to practice law. The same is true of a question so difficult that no one can answer it. Analysis of participant performance on field testing will allow NCBE to determine which questions fall into each of these camps, and which are just right (the “Goldilocks test”), to make the necessary adjustments so that each test item will contribute meaningfully to a total exam score.
2. Whether test items are shown to be fair to all examinees. Although test items should distinguish between those who are ready to enter the practice of law and those who aren’t, they should not make other kinds of distinctions, such as between examinees of different races, ethnicities, socioeconomic backgrounds, genders, or disability status. Field testing will provide data that allows us to ensure that an examinee’s ability to respond correctly to test items doesn’t change based on these irrelevant factors.
3. NextGen exam scoring systems and test delivery. NextGen test item drafters and reviewers are developing scoring rubrics for each test item in concert with exam graders retained by jurisdictions. In field testing, NCBE will work with jurisdiction graders to give them the opportunity to grade the new integrated test items. Field testing will also allow us to try out a new exam delivery model. Although the exam will continue to be administered at jurisdiction-managed exam sites, the delivery method will no longer be paper-based. The NextGen exam will be a fully computer-based exam—­computer delivery will be from a single platform that NCBE will provide. As part of field testing, NCBE will work with jurisdiction admissions authorities to administer the exam under this new model.

In addition, we expect that field and prototype testing will give jurisdictions and law schools the opportunity to see a full-length exam administered as it will be beginning in July 2026.5 The field and prototype exams will look and feel exactly like real exams and will help complete the final step of bringing a new bar exam into a reality: standard setting to help jurisdictions determine their new passing scores.

As a reminder, though NCBE develops the bar exam, it doesn’t determine what constitutes a passing score on the exam. Instead, each jurisdiction sets its own criteria for what constitutes a passing performance. On the current Uniform Bar Exam, jurisdiction-set passing scores range from 260 to 280. Jurisdictions consider a range of different issues and information when setting passing scores, but at the heart of this process is what’s known as a standard-setting exercise. For the NextGen exam, participant performance on the prototype test will be used to guide the standard-setting process for jurisdictions.6

Conclusion

The research testing described here plays a vital role in transitioning from plan to reality for the next generation of the bar exam. Each round of testing will provide the data, information, and insight to ensure that the new exam provides a fair and meaningful assessment to support jurisdictions in determining whether their examinees are ready to begin practicing law.

To stay up to date on development of the future bar exam, subscribe to the NextGen website at nextgenbarexam.ncbex.org/subscribe.

Contact us to request a pdf file of the original article as it appeared in the print edition.