The NextGen Bar Exam: Quarterly Update: Raising the Bar on Score Validity with Double Grading

This article originally appeared in The Bar Examiner print edition, Spring 2025 (Vol. 94, No. 1), pp. 56–59.By Kara Smith McWilliams, PhD

A fair, accurate, and defensible exam scoring process is just as critical as the bar exam itself. In high-stakes testing, every score must reflect a candidate’s true proficiency, free from bias, inconsistency, or un­­necessary stress. That’s why the NextGen bar exam will go through rigorous double grading, where two jurisdiction graders review every constructed response independent of one another, with adjudication (a resolution process when two graders’ scores disagree beyond established thresholds) as needed, ensuring that every such exam item gets evaluated with precision and fairness.

Why Double Grading Matters

Constructed-response questions require evaluative judgment, distinguishing them from multiple-choice items. Double grading enhances reliability by reducing score variability, promoting a consistent and fair evaluation process. Also, standardized training and adjudication protocols support score portability across jurisdictions, reinforcing the integrity of candidate outcomes. NCBE’s double-grading process ensures responses are scored in alignment with clear rubrics, preventing any single grader’s input from disproportionately affecting a candidate’s final score. When two graders disagree beyond an established threshold, the response goes to a trained team leader for adjudication. This structured approach mitigates inconsistencies and enhances scoring accuracy. Without rigorous grading protocols, small inconsistencies can have outsized consequences. By refining rubrics, grader training, and scoring processes, NCBE drives greater fairness, reliability, and confidence in the NextGen bar exam.

Technology-Aided Efficiency

Double grading strengthens score validity, but it must also be efficient and scalable. That’s why NCBE has partnered with an experienced provider to implement a cutting-edge grading platform that automates adjudication tracking, ensuring discrepancies are systematically reviewed and resolved; facilitates seamless grader collaboration, allowing team leaders to mentor graders when inconsistencies arise; and provides integrated scoring guides, giving graders real-time access to rubrics for more consistent evaluations. By leveraging modern technology, NCBE ensures rigor without overburdening graders or jurisdictions, streamlining the entire assessment process.

Improved Grading Consistency: Lower Adjudication and Higher Validity from Field Test to Prototype Test

A key indicator of assessment quality is a low adjudication rate, meaning that when two graders evaluate a response, their scores are closely aligned. Another indicator of quality is alignment to validity papers—pre-scored responses embedded in the grading process to measure grader score accuracy. When adjudication rates decrease and alignment to validity papers increases, it signals that items, rubrics, and grader training are improving, leading to a more reliable and efficient scoring process.

The October 2024 prototype exam provided valuable insights into the effectiveness of our scoring approach. Adjudication rates decreased, demonstrating an increase in inter-rater reliability, while alignment to validity papers improved—both showing that enhanced training and that refined grading materials optimized the process. Graders expressed confidence in the process, reporting that double grading reduced pressure and reassured them that candidates received fair, unbiased scores.

Adjudication Optimization

The overall adjudication rate dropped from 39% in the field test to 25% in the prototype test, a significant improvement in grading alignment. One key scoring component saw adjudication decrease from 36% to just 7%, demonstrating that training refinements and clearer rubrics drive results.

Scoring Validity Optimization

Prototype graders aligned with validity papers 90% of the time, on average. Individual question accuracy improved significantly, with most prototype test questions achieving 90% or higher accuracy. Scoring consistency for NextGen’s most innovative item types, item sets, also improved. Counseling item set scoring consistency increased from 77% to 90%, while drafting item set scoring improved from 76% to 84%, showing that enhanced rubrics and training are working.

These data-driven improvements affirm NCBE’s commitment to ensuring graders are well prepared, scoring protocols are effective, and candidate scores are accurate and fair, setting a new standard for assessment accuracy.

What Graders Say

Current jurisdiction bar exam graders graded the prototype exam. A total of 118 volunteers from 47 jurisdictions participated in this process. NCBE actively gathered grader feedback to refine our approach. They appreciated having a second review, noting that it reduced pressure and ensured fairness. Team leaders valued the adjudication process, as it resolved discrepancies efficiently and equitably. One grader shared that the double-grading process gave them confidence that every candidate’s response was evaluated fairly and accurately. Knowing another expert validated their work and that adjudication was available made the process feel rigorous yet supportive. By listening to graders, NCBE continues to enhance scoring accuracy, training methods, and the grader experience.

The Future of NCBE’s Grading Approach

Ensuring valid, reliable, and fair scores remains central to NCBE’s mission. As we continue to optimize our grading processes, we are exploring enhancements designed to maintain rigorous scoring while improving efficiency. One such approach is proportional double grading, i.e., double grading only a subset of questions, which could reduce the load on graders but maintain validity checks on grading. Additionally, we are conducting artificial intelligence (AI)–assisted scoring research to explore how AI models can provide secondary support to human grading through data-driven insights while preserving the integrity of expert judgment. NCBE is committed to carefully evaluating these advancements and collaborating with jurisdictions to ensure that any refinements align with their needs and expectations.

As we move toward the January 2026 beta test, a multijurisdiction prelaunch test with 1,500 examinees that will replicate operational launch, and the full NextGen bar exam debut, NCBE remains focused on expanding research into scoring validity and reliability, further optimizing our grading platform for efficiency and accuracy.

Our Commitment to Jurisdictions and Candidates

Double grading is more than a process; it is a commitment to fairness, accuracy, and consistency. NCBE leads by balancing rigor with efficiency, leveraging technology, research, and human expertise to deliver high-quality, trustworthy scores. Candidates, bar administrators, and jurisdictions deserve a system that is both innovative and fair, and that is exactly what we are building.

Contact us to request a pdf file of the original article as it appeared in the print edition.