Quarterly Update: The Evolution of the Bar Exam: An Update from NCBE’s New Chief Product Officer

This article originally appeared in The Bar Examiner print edition, Fall 2024 (Vol. 93, No. 3), pp. 32-34.By Kara McWilliams, PhD

As the new Chief Product Officer for NCBE, I’m thrilled to introduce myself to the Bar Examiner community. This role presents an exciting opportunity to develop a holistic product strategy that meets the needs of the jurisdictions and candidates we serve and to lead NCBE’s work in having the bar exam reflect the knowledge and skills the next generation of lawyers will need.

My career has centered on blending psychometrics, assessment design, and product management with advanced technology, always with a focus on research-driven methods to enhance the user experience. I’m honored to bring that perspective to NCBE.

I look forward to providing regular updates here. In today’s column, I’ll share the progress we’ve made on the NextGen bar exam, provide an overview of our recent prototype test, and describe our ongoing research efforts that will shape the candidate experience and ensure the ongoing high standards NCBE is known for.

Progress on the NextGen Bar Exam

Since joining NCBE, I’ve worked with Jake Vizner, our new Chief Information Officer, and the entire NextGen team to drive several key initiatives aimed at supporting both candidates and jurisdictions in their NextGen experience. We have ambitious plans to develop essential infrastructure for the exam, each tailored to address stakeholders’ needs. The following projects are in active development:

• Jurisdiction portal: Aiming to streamline data management and enhance collaboration, this portal will allow jurisdictions to access and manage candidate information, communicate directly with NCBE, and gain tools to support exam administration.
• Candidate portal: This portal will centralize and simplify the candidate experience, enabling examinees to manage their exam process, from registration to results, all in one user-friendly platform. We want to empower candidates with greater access and transparency throughout their bar journey.
• New scoring platform: To improve grading efficiency, we’re working on a modernized scoring platform that will allow for faster, more accurate results. Jurisdictions will experience smoother workflows and quicker turnaround times, which aims to benefit all involved.
• Iterative testing of exam content and experience: From item types to digital usability, we’re conducting iterative testing to ensure the content, format, and overall experience meet the needs of legal candidates and jurisdictions alike. Our recent prototype test, for example, provided crucial feedback for further enhancements.

These projects are all in various stages of development, and we’re taking a deliberate, research-­focused approach to ensure each component is fully optimized and validated before going live in July 2026.

A Comprehensive Prototype Test of the NextGen Bar Exam

One of our most significant milestones this year has been the NextGen prototype test, an extensive and highly detailed simulation of the full exam experience. The prototype test involved over 2,000 participants across 36 test sites in 32 jurisdictions, making it one of the most extensive trials in NCBE’s history. All participants completed the July 2024 bar exam, enabling us to draw comparisons between the two exams. This test was essential for evaluating how well the new exam structure and content align with our goals. Four key areas of focus for this test included:

1. Blueprint of the full assessment: The prototype test included a full, nine-hour exam. The blueprint represents items to measure the knowledge and skills newly licensed attorneys need and includes innovative items specifically designed to assess practical skills.
2. Prototype platform experience: Participants took the test on a prototype of the digital delivery platform that will be used for the actual exam. Observing participants in this environment allowed us to gather feedback on usability and refine the platform to ensure a smooth, intuitive experience.
3. End-to-end exam journey: Testing the entire candidate journey was essential to identify any friction points and ensure a seamless user experience, from login through submission. This process allowed us to refine our approach and align it more closely with the needs of candidates and jurisdictions.
4. Grading platform for constructed responses: The prototype test includes the initial trial of a new grading platform designed to evaluate constructed responses accurately and efficiently. With this platform, jurisdictions will benefit from a streamlined grading process and enhanced tools and capabilities.

This large-scale prototype testing allowed us to assess both operational feasibility and user experience. The jurisdictions, too, benefited from this experience, as they had the chance to trial the entire process and provide feedback on everything from administration to logistical details.

With complete data from all participants, we can now move forward with essential research. This includes analyzing items to gauge the quality of individual questions, establishing a base scale for the new exam, conducting standard-setting studies, and developing a concordance table to help jurisdictions understand how current bar exam scores align with NextGen scores. The insights gained will ensure the NextGen bar exam is both fair and rigorous, meeting NCBE’s high standards of quality.

Ongoing Testing and User Research for Continuous Improvement

The prototype test represented a critical step, but our work is on­­going. We’re committed to continuous improvement through a comprehensive user experience research (UXR) study. For this study, we’ve distributed surveys to all participants and administrators involved in the prototype test, and we’re conducting focused discussions to gain deeper insights.

Key elements of the UXR study include:

1. 10 focus groups and 20 interviews with administrators: Through these sessions, we are gathering feedback on the exam administration process, with insights that will directly inform improvements to exam logistics and functionality.
2. 6 focus groups and 18 interviews with participants: Participant feedback is essential for understanding the user experience. These sessions will help us further optimize the platform and refine any areas identified as needing enhancement.

The insights we gain from this research will drive our iterative testing and improvement efforts over the next several months, ensuring the NextGen bar exam meets our quality and reliability standards.

The Path Ahead

I’m incredibly proud to work with an organization so dedicated to ensuring fairness, quality, and innovation in its assessment products. The journey to the NextGen bar exam is guided by rigorous research and a commitment to excellence. I look forward to sharing more updates as we move closer to launch, confident that the exam will reflect both the needs of the legal profession and NCBE’s standards of excellence.

In the next installment of this column, I’ll provide further insights from our prototype test results and our future plans. Thank you for being a part of this journey with us as we bring this ambitious vision to life.

Contact us to request a pdf file of the original article as it appeared in the print edition.