The Next Generation of the Bar Exam: Quarterly Update

This article originally appeared in The Bar Examiner print edition, Winter 2021-2022 (Vol. 90, No. 4), pp. 31–33.By Kellie Early

Steady Progress on the Next Generation Exam

NCBE is progressing on several fronts with the implementation of the next generation of the bar exam. This update provides an overview of our progress since the fall edition of the Bar Examiner was published.

Scope of Content to Be Assessed

NCBE’s Content Scope Committee has completed its work to delineate the scope of coverage of the eight Foundational Concepts and Principles (subjects) and the seven Foundational Skills that will be measured on the new exam to ensure it carefully aligns with minimum competence for entry-level practice. The Content Scope Committee was composed of a diverse group of 21 dedicated legal professionals, including bar examiners, legal educators, deans, and practitioners. The committee’s recommendations were compiled in voluminous individual reports, one on each subject and one on the skills, capturing the committee’s discussion about the specific topics that are important to entry-level practice and the depth of knowledge new practitioners need when those topics arise. The committee also identified the specific lawyering tasks that would allow examinees to realistically demonstrate, in the context of a written bar exam, their competence in the Foundational Skills.

Scope Outlines Published for Comment

The Implementation Steering Committee reviewed the Content Scope Committee’s reports in March 2022, and scope outlines were prepared to publish for stakeholder comment. To optimize the comment process and focus the feedback we receive, the scope outlines simply identify what will be assessed on the new exam but do not include the detailed discussions surrounding the rationale for the recommendations. The scope outlines are to inform stakeholders of the intended scope of coverage and to gather input on whether there are any significant oversights in terms of the topics or the lawyering tasks to be assessed. Notice was sent to jurisdictions, law schools, NextGen website subscribers, and other stakeholders when the scope outlines were published for comment.

Next Steps: Test Content Specifications

There remains significant work to be done and the scope outlines are only the first step in developing the more detailed test content specifications, or “blueprint,” for the new exam. The current plan is for the test content specifications to identify the sources of law for the topics being tested, specify the weighting or emphasis of the subjects and skills to be assessed, provide annotations about what is covered within subjects or topics, and incorporate sample test questions to provide examples of how the knowledge and skills may be tested in an integrated design, among other possible features. The goal for the test content specifications is to ensure the content validity and fairness of every test form by providing a blueprint to those drafting and assembling the exam and information about what to expect to those preparing to take the exam. To learn more about the role of test content specifications, see the April (nextgenbarexam.ncbex.org/the-work-begins-developing-content-specifications-for-the-nextgen-bar-exam/) and August (nextgenbarexam.ncbex.org/the-what-and-why-of-test-content-specifications/) 2021 blog posts.

Test Development

NCBE’s drafters and staff are experts at producing high-quality multiple-choice, essay, and performance test questions that measure specific knowledge and skills that new lawyers should be able to demonstrate to become licensed. Creating an integrated exam that tests knowledge and skills holistically provides many possibilities for the redesigned exam, and we have been developing prototypes of new question types. Developing item sets, which are a collection of test questions based on a single scenario or stimulus, and questions centered on the new skills we have not measured previously on the bar exam has been the initial focus of our prototyping. We aim to pilot test prototype questions on a small scale later this year to see how they perform. Pilot testing involves administering sets of test questions to diverse groups of law students under semirealistic conditions to evaluate how the questions perform and obtain feedback from the students. Pilot testing will be followed next year by field testing, which will involve administering more questions to larger groups of “pseudo-examinees” under realistic testing conditions. Diversity, fairness, and inclusion considerations are key in our test development efforts and will be a focus of the pilot and field testing processes to ensure the new exam continues our mission to fairly assess all candidates. One example of an issue we will be evaluating during pilot and field testing is the effect of providing legal resources (e.g., Federal Rules of Evidence) for candidates to use during the exam, including the optimal way to provide such resources for candidates to use efficiently in the time allotted.

Computer-Based Test Delivery

The new exam will be delivered on computers rather than paper, and NCBE has been exploring the two options of administering the new exam: at computer testing centers or on examinees’ laptops at jurisdiction-managed sites. There are advantages and drawbacks to each of these options. Although the use of testing centers would provide more uniform testing conditions and computer equipment, it would also require a larger bank of test questions to produce comparable test forms over the longer testing window necessitated by seating capacity constraints. The use of laptops at jurisdiction-managed sites would permit us to continue the current “event-based” administration schedule, which does not require as large of a question bank, but it provides less uniformity of testing conditions and equipment. We are also sensitive to the challenges some jurisdictions increasingly face in securing testing venues and proctors. These are just a few of the many factors we are continuing to evaluate before reaching a decision on this issue. We appreciate the input we have received from the jurisdictions and continue to welcome it.

Grading and Scoring

As NCBE develops the next generation of the bar exam, we are considering options for grading and scoring that are best suited for an integrated design structure. The new exam will be composed of a mix of item (question) formats, including constructed response items. The items, some of which will be combined in item sets, may measure knowledge and several skills, so they will potentially be scored on multiple dimensions. With this type of design, centralized grading for the constructed response portion is important to enhancing score reliability, so NCBE is exploring opportunities to further support consistency. Jurisdictions would continue to provide qualified graders, and NCBE would provide a grading platform, training, scoring rubrics, and support to ensure uniform application of scoring rubrics. Calibration sessions would be held prior to operational grading, and monitoring and real-time feedback would be provided to graders via the grading platform throughout the process. The centralized grading process would be developed with input from jurisdictions and designed to ensure that scores can still be reported within a reasonable time frame from the administration of the bar exam. NCBE will begin exploring grading options in more depth in the coming months.

Jurisdiction Advisory Committee

The Jurisdiction Advisory Committee (JAC) had its kickoff meeting in February 2022. The JAC will help NCBE to consider input from all jurisdictions, and from law schools and candidates, on various issues to ensure stakeholder perspectives are considered as we implement the new exam. At the inaugural meeting, the JAC members were invited to share what they were most excited and concerned about regarding transitioning to the new exam. Among the things that most excite them are the prospect of bringing more uniformity to the bar exam administration and grading processes; the transparent and research-based process that NCBE has followed and continues to follow in redesigning the bar exam; the possibility of reducing testing time from the current 12 hours, assuming the necessary reliability and validity of scores can be achieved in a shorter exam; and the benefits of administering the exam at computer testing centers. Members’ main concerns included using computer testing centers and centralized grading (topics that cause both excitement and trepidation), as well as the time needed to amend rules or seek statutory changes, what the new exam will cost, and when the Multistate Bar Exam will no longer be offered by NCBE. The topics echoed what we heard from jurisdictions at the UBE Forum in January, and we will work with the JAC to have in-depth conversations around each specific area of concern as we make important decisions.

Recent and Upcoming Events

This past fall and winter, NCBE made presentations and answered questions about the NextGen exam at the AccessLex LexCon conference in November, and the Association of American Law Schools (AALS) Annual Meeting and the UBE Jurisdictions Forum in January. In addition, we were asked to speak at December meetings of the California Blue Ribbon Commission and the Minnesota State Board of Law Examiners, and at meetings in February of the Texas Board of Law Examiners and the Massachusetts Supreme Judicial Court Steering Committee on Bar Admissions and its subcommittees.

This spring, we will be making presentations at the National Association of Law Placement (NALP) Annual Education Conference, NCBE’s Annual Bar Admissions Conference, and the Association of Academic Support Educators (AASE) Annual Conference. We look forward to seeing some of you at these events. We will continue to keep everyone informed of our progress through our website, so please subscribe if you haven’t already! Subscribe here: nextgenbarexam.ncbex.org/subscribe/.

Subscribe to the NextGen website at nextgenbarexam.ncbex.org to stay informed as our work progresses and learn of opportunities to participate.

Contact us to request a pdf file of the original article as it appeared in the print edition.