Texas Bar Exam: 2023 Standard-Setting Study

This article originally appeared in The Bar Examiner print edition, Winter 2023-2024 (Vol. 92, No. 4), pp. 20–22.

By Nahdiah HoangIn June 2016, the Supreme Court of Texas created a task force to study the Texas Bar Exam and make recommendations to the Court. Among the issues the task force was charged with studying were scoring policies and bar exam components, including whether Texas should adopt the Uniform Bar Examination (UBE).

In May 2018, the Supreme Court of Texas Task Force on the Texas Bar Exam recommended that Texas adopt the UBE and that if the UBE were adopted, a standard-setting study be conducted after three years to evaluate whether the ­passing score had been appropriately set.

After receiving the Task Force Report and evaluating public comments, the Court issued an order in October 2018 adopting the UBE and setting the minimum passing score at 270, which was roughly equivalent to the minimum passing score of 675 out of 1,000 on the then-­current Texas Bar Exam.

Texas Board of Law Examiners (BLE) staff administered Texas’s first UBE in February 2021. Because of the COVID-19 pandemic, the exam took place remotely. Five in-person UBEs followed, from July 2021 through July 2023.1

In keeping with the Task Force’s recommendation to conduct a standard­-setting study after three years, we (BLE staff) planned to conduct a study after the July 2023 exam.

Determining the minimum passing score on a bar exam is an important decision that involves both the science of psychometrics and the art of policymaking; as staff for the Texas BLE, our role was to provide policy­makers with context and arrange for experts to provide the psychometric perspective.

Planning

In 2021, we began reviewing studies performed in California2 and Oregon,3 researching potential consultants to assist in the process, developing a budget, determining the applicability of state procurement requirements, and creating a timeline with the goal of conducting the study in fall 2023.

In January 2022, we presented the board with a proposed timeline and contracted with a consultant. After several meetings with the consultant and NCBE, we decided on the study’s format:

• Panelists would review a subset of MBE questions, all six MEE questions, and both MPT items from the July 2023 Texas Bar Exam.
• Panelists would attend an introductory Zoom meeting and an in-person meeting over two and a half days.
• Up to 30 panelists would participate, allowing the use of two groups during the in-person meeting.

With this format in place, we presented a budget request to the board in June 2022.

Selecting Panelists for the Study

The consultant provided guidance on who should serve as panelists: Texas attorneys who were familiar with what a new lawyer should be expected to do, either because they had recently been a new lawyer themselves or because they supervised or worked with new lawyers.

We explored several methods for selecting panelists. One option was to work with the State Bar of Texas to identify and contact relevant attorneys. Another option was for board members to select panelists. We settled on having each board member suggest at least 10 potential panelists; BLE staff then assembled a panel from those suggestions with an eye toward a diverse panel in terms of sex, race, geography, and firm size.

We covered travel, lodging, and most meals for all panelists, and the State Bar of Texas agreed to award the panelists significant mandatory Continuing Legal Education credits.

Study Execution

The study was conducted in Austin, Texas. Twenty-seven panelists participated. Several observers attended, including BLE staff and board members, a law school assistant dean, and NCBE representatives.

On the first day, all panelists met together in person. After orientation and training, they reviewed 70 MBE questions from the July 2023 exam and, for each question, assessed whether a minimally qualified candidate should be able to answer it correctly. After discussion, panelists reviewed their assessments and made any changes they deemed appropriate.

On the second day, the panelists split into two groups to review the MEE questions. Each group reviewed four of the six questions, which meant two questions were reviewed by both groups. (This overlap provided information on consistency between the two groups.) For each question, we provided the panelists with the MEE question, the grading materials used by our graders, and 10 randomly selected answers that had received scores ranging from 1 to 6. Each panelist reviewed all 10 answers and selected the one that they believed represented the performance expected of a minimally qualified candidate. After discussion, panelists could change their selection.

On the third day, the panelists broke into the same two groups from the previous day to review the MPT items. Each group reviewed one MPT. The process was the same as for the MEE questions—the panelists received the MPT item, grading materials, and 10 answers. The panelists reviewed all 10 answers and selected the one that represented the performance expected of a minimally qualified candidate. Again, after discussion, they could change their selection.

The consultant gathered information at each stage of the study, used insights from examinee performance on the July 2023 exam (including the actual scores given on each MPT and MEE question used in the study) and calculated the panel’s recommended passing score along with standard error above and below the recommended score. (Standard error is a statistic that quantifies the degree of variability for one sample and is often called the margin of error. The standard error is related to the standard deviation, which is the measure of the spread of scores—that is, the average deviation of scores from the mean score, both above and below the mean score.) The panelists discussed the recommendation, and offered individual, written assessments of their confidence in the study.

Context: Standard Setting in Other Jurisdictions

By the time the consultant presented their report for the January 2024 board meeting, our board had met with the consultant in person twice, and several board members had observed part or all of the standard-setting study. The board members had also reviewed studies other jurisdictions had performed. They understood the study’s mechanics and purpose, they understood false positives (where an examinee who is not a minimally qualified candidate still passes) and false negatives (where a minimally qualified candidate fails), and they were aware of the relevant policy considerations.

Also of interest, by the time the board met in January 2024, every jurisdiction that had once had a minimum passing score above 270 had lowered its score to 270. To provide the board with additional context, we informed them of the recent changes and addressed the history of score changes in other jurisdictions.

Oregon, for example, conducted a study in May 2021 designed to review its then-existing passing score (274) rather than to recommend a passing score, so Oregon’s methodology differed slightly from ours. (Oregon selected eight examinees who scored between 264 and 278, reviewed each complete exam, and made a holistic assessment of whether a given examinee had demonstrated minimum competence.) Oregon’s task force determined that the minimum passing score should be between 268 and 273, and “all members of the Task Force agreed that 270 is the score that may come closest to balancing the ideals of the minimum standard, as well as achieving consistency with other Western states, while still adequately addressing the concerns of members on various policy issues and risk tolerances.”4 Oregon changed its score to 270 for the July 2021 bar exam.

By February 2023, Alaska and Colorado had changed their scores to 270 from 280 and 276, respectively.

Also in February 2023, Delaware conducted a standard-setting study using a methodology very similar to the one we used in Texas. Delaware reviewed MBE questions, MPT items, and Delaware essays questions (not MEEs). The study resulted in a mean recommendation of 144.8 as the minimum passing score (Delaware’s score was 145 at the time), and, after considering the study and various policy issues, Delaware changed its minimum passing score to 143 beginning with the July 2023 exam.5

By July 2023, Arizona and Idaho had changed their scores to 270 from 273 and 272, respectively, and Utah had changed its score from 270 to 260.

And in December 2023, Pennsylvania announced it was changing its score from 272 to 270.

The Texas BLE will continue to deliberate the results of the standard­-setting study in their upcoming meetings.

Contact us to request a pdf file of the original article as it appeared in the print edition.