Shhh . . . California Examinees May Be Sleeping In after Day Two of the Bar Exam

This article originally appeared in The Bar Examiner print edition, June 2017 (Vol. 86, No. 2), pp 30–35.

By Patrick R. Dixon and Alan S. Yochelson

On July 25 of this year, thousands of applicants will take the first day of the California General Bar Examination.

On July 26, they’ll take the second day of the exam.

On July 27, they’ll sleep in—and that will be a new day in California.

For many decades, the third day of the California bar exam has been anything but a day of rest for the test takers, who have had to spend all or part of the day completing their work. But this year, for the first time, the exam will be given in just two days.

The California bar exam, in the three-day format that was used for the final time in February 2017, consisted of

• six one-hour essay questions,
• the six-hour, 200-question, multiple-choice Multistate Bar Examination (MBE), and
• two three-hour performance test questions.1

In July, in the first format change in decades for the California exam, applicants will be faced with

• five one-hour essay questions and one 90-­minute performance test question on day one and
• the MBE on day two.

Officials of the State Bar of California studied the change carefully and concluded that the new format would maintain test reliability, be fairer, and save money. They also noted that the purpose of the test is strictly to determine whether the applicant possesses the minimum tools required to practice law. “It’s a test of minimum competence, not stamina,” said Gayle Murphy, Senior Director of the State Bar’s Office of Admissions.

History of the Composition and Grading of the California Bar Exam

The California bar exam is widely considered to be the most difficult in the nation, primarily because it has the second-highest cut score, behind Delaware.2 Some 13,000 applicants take the exam annually, with a historical combined annual pass rate oscillating around 50 percent. In 2008 the July pass rate hit a peak at 61.7 percent, but then hovered around 55 percent for five years before plummeting from 55.8 percent in 2013 to 43 percent in 2016. Officials say that the new exam is not intended to alter these numbers. Currently, it is expected that the cut score will remain at its high level, but it is under intense study and scrutiny by many different stakeholders, and it is possible that there may be some adjustments in the future. Moreover, a key requirement in the adoption of the two-day format was that there would be no loss of reliability in the exam.

The 1970s: MBE Adoption, Changes in the Number of Essay Questions, and the Institution of Reappraisal Procedures

For 40 years leading up to 1972, the California bar exam consisted of some 20 essay questions and was administered over three days.3 To pass, an applicant had to score at least 70 percent of the total points possible. In 1972, the exam was extensively revised: the number of essay questions was trimmed to 12, the MBE was added, and a reappraisal procedure was instituted for applicants who scored near the cut score. The essay questions were administered in three batches of five questions each, with applicants instructed to answer any four of the five questions in each batch (thereby totaling 12 questions) and given 3.5 hours to complete each batch (i.e., 52.5 minutes per question). Each essay was worth 100 points, and on average there were four to five graders (called “readers” in California) scoring each question.

The MBE was weighted to be worth 514 points, or 30 percent of 1,714 total possible points. This was done in part because it made it at least theoretically possible for an applicant to reach 70 percent on the strength of the essay score alone; even with a zero score on the MBE, an applicant could pass the exam with a perfect 1,200 on the essay questions. (The point was probably more psychological than practical because, of course, in reality this would not occur.)

Reappraisal was instituted for applicants who scored 1,140 to 1,199 points, or 66.5 percent to 69.9 percent of the total possible. Exams scoring in this range were reviewed independently by two reappraisers who each rendered a pass/fail decision. If the first two reappraisers split their verdicts, a third reappraiser reviewed the exam and made the final decision.

This structure was refined in a series of changes that took effect in 1978. The number of essay questions was dropped again, from 12 to 9 (with no choice of questions), and the time to answer each question was increased to 60 minutes. Weight was added to the MBE so that it became worth 600 points, or 40 percent of 1,500 total possible points, compared to the 900 possible on the essay questions. Also, a more nuanced grading system was adopted that concentrated readers’ time and effort on applicants near the cut score. The first grading pass included the MBE and one randomly selected set of three essay questions. If the applicant’s score on the first read was high enough, he or she would be passed without reading the other six essays. (The practice of passing some applicants without reading all of their essays at least once was eventually discontinued with the July 1984 examination, as it was found to be no more expensive to read all of every applicant’s answers at least once.) If the applicant’s score on the first read was not high enough to pass, the rest of the essays were read, and those applicants with total scores of 71 percent or higher were passed. Those who scored between 67.3 percent and 71 percent went to reread, and those who scored below 67.3 percent failed. The 67.3 percent threshold was established after a study of past exams, to include all that had even a small chance of passing on reread and reappraisal. In reread, the essays were graded by a different reader and the average of the applicant’s two essay scores was added to the MBE score. Those with a reread score of 70 percent or above passed, those below 68.7 percent failed, and those in between went to reappraisal. In reappraisal, the MBE and essays were all reviewed by a single reappraiser (one reappraiser per grading team) who rendered a pass/fail decision.

The 1980s: A Short-Lived “Bifurcation Rule,” Addition of the Performance Test, and Scoring Adjustments

In 1981, the State Bar implemented a “bifurcation rule”: applicants who failed the exam but scored 70 percent or more on either the MBE or the essay questions could pass a subsequent exam by improving the lagging score to at least 70 percent. Applicants in this category could repeat either the entire exam or just the section they needed to improve. The bifurcation rule was withdrawn in 1983 because it was shown to have negatively affected the pass rate, which was not intended, and because of the addition of the performance test section to the exam, which would have rendered the rule too complicated to implement.

Meanwhile, in three exams in 1982 and 1983, two of the nine essay questions were expanded to 90 minutes each instead of 60, with the two longer questions composing a single, 180-minute test session. The expansion of two of the questions to 90 minutes was discontinued when it was shown to have no effect on passing rates.

In July 1983, the performance test (PT) section was added to the California exam and the number of essay questions reduced from nine to six.4 This was done in part because the State Bar wanted to encourage law schools to include more clinical legal education in their curricula. Initially, each PT problem included both multiple-choice questions and an essay question, with the scores on the ­multiple-choice component scaled to the MBE. The exam had a new maximum score of 1,800 points: 600 each for the MBE, the now six essay questions, and the PT, with 200 of the PT points coming from the multiple-choice questions. The passing score was still 70 percent, or 1,260. The multiple-choice portion of the PT was discontinued in 1985, after which the two PT questions were written only and were still worth 600 points total. The grading system adopted in 1978 was adjusted to the new exam framework, so that scores between 1,211 and 1,278 went to reread, and reread scores between 1,237 and 1,260 went to reappraisal.

In 1987, the scoring was revamped to scale the written raw scores (essays plus PTs) to a distribution with the same mean and standard deviation as the MBE scores, and then to compute a total score with 2,000 possible points, with 65 percent of the weight given to the written components and 35 percent to the MBE. The passing score was set at 1,440, which was the average score that would have produced the same pass rate on the 20 previous exams given between 1977 and 1986 had they been scored according to the new system, with a slight adjustment to account for the reread-reappraisal regime.

2006: Revision of the Reappraisal Procedure

The final noteworthy change in the California bar exam before this year came in 2006, when the reread-reappraisal protocol was revised. Now, after reread, if the first and second readers’ scores for an answer differ by fewer than 10 points, the average stands as the final score. If they differ by more than 10 points and the applicant has a failing total score because of the divergent reread score, the grading team leader reviews the answer and assigns a final grade that can be higher but not lower than the ones given by the readers.

Reducing the California Bar Exam to Two Days: Past and Current Efforts

In a 2005 report summarizing the evolution of the California bar exam over the decades, psychometrician Stephen Klein, Ph.D., wrote, “There have been many changes in the GBX’s [General Bar Examination’s] pass/fail rules, structure, format, the weights attached to its sections, and the number of readers grading the answers to the written questions . . . . Nevertheless, GBX passing rates have generally risen and fallen in close harmony with variations in the candidates’ mean LSAT and MBE scores . . . . Thus, the standards for passing have remained fairly stable over time.”5

A Proposal for a Two-Day Exam in 2001 Is Rejected

Of course, one major change that was not made was a shortening of the exam so that it could be completed in two days instead of three. But that wasn’t for want of trying. In 2001 the Committee of Bar Examiners discussed a proposal to reduce the California exam from three days to two with the State Bar’s Board of Governors. The exam that the Committee recommended then was significantly different from the two-day exam that will be given this July. The exam proposed in 2001 closely resembles the current Uniform Bar Examination in use in over half of U.S. jurisdictions. The 2001 proposed exam included six 30-minute essay questions, a three-hour performance test, and the MBE. After discussing the proposal and questioning members of the Committee about it, the Board voted to reject the two-day exam and to continue to test bar applicants using the three-day exam. The feeling among the governors was that the proposed 2001 two-day exam did not sufficiently test applicants’ writing skills.

Revisiting the Possibility of a Two-Day Exam in 2011: A Study Is Commissioned

A decade later, the State Bar decided to revisit the possibility of a two-day format. In 2011, the State Bar commissioned a study by psychometrician Klein and his colleague, Roger Bolus, Ph.D., on the estimated effect on the quality and pass rates of the exam if it were to be modified; the results of the study were submitted in December 2011.6 Klein and Bolus studied the results of 20 exams administered from 2001 to 2010, totaling more than 125,000 applicants. Using the historical data, the pair looked at how score reliability was affected by different numbers of readers, and how it would be affected by increasing the weight of the MBE from 35 percent to 50 percent, and by shortening the written portion of the exam from 12 hours (six one-hour essay questions and two three-hour PTs) to 6 or 7 hours. They also looked at the effect of these changes on the pass rates of ethnic and gender groups. The researchers defined score reliability as “the likelihood that applicants would receive the same score[s] . . . regardless of the particular set of California bar exam essay and PT questions they were asked or the set of readers who graded their answers.”7 In other words, “the higher the score reliability the greater the confidence that can be placed in the results”8 to accurately measure the applicant’s knowledge and aptitude at the time of the exam. Reliability was rated on a 0-to-1 scale, with 1 being most reliable.

With regard to the number of readers, the study found that a second reader increased reliability somewhat, by 0.06 points, but more readers beyond two had much less of an effect. The psychometricians concluded that the numbers supported California’s system of giving a second read to exams near the cut score, but that the written components’ variance from perfect reliability was due mainly to “an interaction between applicants and questions rather than from differences among readers in the scores they would assign.” In other words, applicants were better prepared to answer some questions than others, and these relative strengths and weaknesses varied from applicant to applicant.

On the MBE weighting, the psychometricians found that increasing it from 35 percent to 50 percent actually improved reliability by 0.05–0.06 points, and it eliminated the difference in passing rates between men and women. With the exam’s then-current weighting, women as a group had a slight advantage over men, as women tended to do better on the written portion of the examination and men did better on the multiple-choice portion.

On shortening the written portion of the exam, the researchers used a couple of models, one sampling five essay answers and one PT answer, and one sampling three essay answers and one PT answer. Both of these models weighted the written and MBE sections equally. The analysis “shows that reducing test length does not affect overall passing rates or exacerbate the differences in rates that are typically found among racial/ethnic groups.”9 The report concluded, “California can implement a two[-]day exam in a way that improves test quality, maintains existing pass/fail standards, and does so without making it more difficult for minority applicants to pass.”10

2013–2015: Public Comment and Hearings, Final Recommendation, and Adoption

The State Bar’s Committee of Bar Examiners invited public comment and held public hearings in 2013 on the proposed reduction of the exam to two days. Patricia White, chair of the Committee of Bar Examiners in 2014–2015 and a former member of the Bar’s Board of Governors, recalls that “there was some pushback that writing ability is important and therefore the PTs should be kept at the current length.” But, she points out, most of the public comments were supportive. The Committee also met with the state’s law school deans, she says, and all but one expressed support for the change. As stated by Erwin Chemerinsky, dean of UC–Irvine School of Law and dean designate at UC–Berkeley School of Law, “Why put people through an extra day if it’s not going to make any difference [to whether] they pass or not?”

Gayle Murphy, the Senior Director of the State Bar’s Office of Admissions, noted that in addition to maintaining reliability, saving time, and reducing stress, the change could shorten the time applicants have to wait for results (although no promises were made) and definitely will save the State Bar and applicants a fair amount of money. Renting facilities for two days instead of three (and for four days instead of six for applicants needing double-time accommodations to take the exam) and reducing the number of graders could save the State Bar $1 million annually, and this will enable the Bar to hold off for some time on raising the exam fee for applicants.

In March 2014 the Committee recommended the change to the Bar’s Board of Trustees. On the day before the July 2015 bar examination, the trustees voted unanimously to adopt the two-day format, starting with the July 2017 exam, adding a requirement that the new exam be evaluated after two years to identify any problems or anomalies. The following March, the California Supreme Court gave the change the final approval.

So, if you know anyone taking the California General Bar Examination this July, you can call on day three to offer your congratulations on finishing, but don’t call too early—he or she may still be sleeping.

Contact us to request a pdf file of the original article as it appeared in the print edition.