President's Page

This article originally appeared in The Bar Examiner print edition, Spring 2019 (Vol. 88, No. 1), pp 2–5.

By Judith A. Gundersen

February Bar Exam Results and Operation Varsity Blues, or, Do You Want the Good News or the Bad News First?

How About the Good News First? Surprise, It’s the February 2019 Bar Exam Results

There is some good news to report about the February 2019 bar exam results. The national MBE mean was up 1.2 points over last February, from 132.8 to 134.0. That is the first time we’ve seen an increase in the national February MBE mean since 2013. The MBE means in some jurisdictions were down, however, so not all jurisdictions will see an improvement in their pass rates. Pass rates are a function of where the cut score is set and examinees’ demonstrated proficiency on the bar exam components. And in small jurisdictions, a few examinees knocking it out of the park or faring badly can have a big impact on scores and pass rates.

As I’ve written in this column before, jurisdictions receive their examinees’ MBE scores from NCBE, but the jurisdictions themselves grade the written components of their exams. Depending on the number of examinees, the grading process can take a few months. So as I write this, fewer than half of the jurisdictions have completed the process of grading and score combining.

Repeaters v. First-Time Takers—Who Took the February Exam?

Overall, the percentage of likely repeat takers was up again—about 68% in February 2019 (versus 67% in February 2018 and 62% in February 2012). Likely first-time takers comprised about 22% of all examinees, with the remaining approximately 10% unknown because of non-use (or only recent use) of the NCBE Number that doesn’t let us track back far enough on prior testing, or insufficient biographic information to track prior testing with certainty. (Note that the percentage of likely repeat takers and likely first-time takers referred to here is based on NCBE’s data on an examinee having previously taken the MBE in any jurisdiction, which differs from the basis of the repeat/first-time passing percentages provided by the jurisdictions for our annual statistics, which are based on the examinee’s testing experience in the reporting jurisdiction only and do not account for possible previous attempts at the bar exam in other jurisdictions.)

Of those examinees we classify as likely repeaters, we can divide them into two main groups—stronger repeaters (those who have likely previously passed a bar exam) and weaker repeaters (those who have likely previously not passed a bar exam). (About 1% of examinees are likely repeaters who are unknown to be stronger or weaker repeaters.)

Stronger repeaters constituted 6% of all examinees in February 2019, down from about 7% in February 2018.
Weaker repeaters, on the other hand, represented about 62% of the total examinee population in February 2019, up from almost 59% in February 2018.

Certainly, the fact that the performance of all groups improved means that many jurisdictions should see higher pass rates.

How Did the Various Examinee Groups Perform on the February Exam?

In a word, better. But the improvement in MBE scores was not the same across the board. Let’s look at performance among these groups:

Likely first-time takers had a score gain of .6 of a point over February 2018.
Weaker repeaters had a score gain of 1.8 points over February 2018.
Stronger repeaters saw their means rise by about .5 of a point.
Unknown status examinees’ means were up by 2.5 points.

Certainly, the fact that the performance of all groups improved means that many jurisdictions should see higher pass rates.

What does all this mean or portend for July? It’s hard to know because, as is always the case, there is a lot we don’t know about the upcoming examinees: what curricular choices these examinees made, where they went to law school, what kind of bar preparation they undertook, whether they had to work while preparing for the bar exam, and so on.

High Reliability

The reliability of the February 2019 MBE was .92—a value that is the same as that observed in February 2018. (Reliability is an indicator of the consistency of a set of examination scores, with a maximum value of 1.0.) A high-stakes exam like the bar exam—an exam that is part of what determines entry to the profession—should have a reliability of .90 or higher so that examinees and the public can have confidence in the results—that they are reproducible and consistent.

Now the Bad News . . . High-Profile Cheating

It’s unusual that a testing story resonates in popular culture. (Not Aunt Becky!) This time it did. (Not to mention the press coverage of Kim Kardashian West’s announcement of her intention to take the bar exam through law office study!)

The college admissions scandal (code-named Operation Varsity Blues by FBI investigators), which revealed that parents had bribed and cheated their children’s way into top colleges and universities, really gripped me, as I’m guessing it did many of you. Hearing about the scandal made me think that it presented an opportunity to write about the “why” behind a standardized test—that if administered securely and uniformly, such a test is an important part of a fair system to allow examinees to be admitted to a school (or a profession) based on what they know, not who they know or how much money they have.

For those of us with kids who have gone through the college admissions process, there is a personal reaction to the admissions scandal: disgust at the cheating scheme in which these families participated—and the fact that it involved families that already appear to have more than enough yet feel entitled to even more. It is unclear whether we will ever know the extent of the harm that this scheme did to deserving students who were deprived of an admissions slot that instead went to a fake athlete or to a student whose high ACT score was the result of a colluding proctor’s help.

To those who depend on our exams—courts, admissions staff, bar examiners, bar associations, the public, and the examinees themselves—as well as the law schools that prepare students for the exams, we pledge to you that we have done and will continue to do everything in our power to ensure exam integrity and security.The more fundamental harm that I think has been done is to the bedrock notion of fair play in high-stakes admissions decisions. For those of us who develop, administer, score, and routinely defend standardized tests as a fair, objective way to make important admissions decisions, this scandal highlights what can happen when the very test that is designed to eliminate unfairness is compromised through cheating. And in the recent college admissions scandal, the kind of cheating that is alleged is jaw-dropping: falsified accommodations requests to obtain unwarranted extra testing time, collusion with proctors, and manipulation of exam results. If proven to have occurred, these actions mean that the playing field for those seeking admission at the colleges and universities in question was anything but level. For those who rely on standardized tests to make life-affecting decisions about admissions or licensure, this scandal’s revelations strike at the very core of what standardized tests are designed to do—ensure fairness for all applicants.

While the cheating alleged in the college admissions scandal is brazen (not to mention illegal), bar admission administrators could share some examples of cheating on the bar exam that range from inept and obvious to sophisticated and stealthy. Jurisdiction staff members are on the front lines to deter cheating. And the front-line defense and detection landscape has changed as technology has made it possible to cheat in more sophisticated ways. Our Testing staff members and Director of Test Security Beth Hill, with the assistance of outside security professionals, are constantly monitoring the latest developments in cheating. And we are also assisted by technology to better monitor and thwart cheating.

To be sure, the number of examinees who try to cheat is very small, but the consequences to all examinees can be big. That is why jurisdictions and NCBE are very strict about security measures such as insisting on proper identification and screening of examinees, adhering to predetermined seating arrangements, monitoring bathroom breaks, hiring and vetting proctors, and prohibiting personal items in the exam room. NCBE receives a report from each jurisdiction immediately following each exam indicating the presence of suspected cheating incidents. We, along with jurisdiction staff, follow up to investigate suspicious behavior as warranted. On the occasions when cheating is suspected during the bar exam, observational evidence (sometimes from other examinees) and data analytics are used to investigate and confirm if the evidence supports the suspected cheating. For examinees who are found to have attempted to cheat on the bar exam, the consequences are severe and long-lasting: cancellation of scores, character and fitness hearings, and postponement or even outright denial of the right to be admitted to practice law. There may be legal consequences as well for copyright violations. The college admissions scandal has underscored what we all know: cheating can happen, and strict measures must be in place to prevent and detect it.

The gaming of the accommodations process could probably be a topic in its own right for this column. NCBE and admissions authorities have seen bar exam accommodations requests that are clinically or medically unsupported and apparently not made in good faith. Again, like examinees who attempt to cheat on the exam itself, examinees who unfairly seek accommodations constitute a very small proportion of all examinees who request appropriate accommodations. The great majority of examinees seeking accommodations have the supporting documentation and a demonstrated need for such accommodations. NCBE is committed to providing reasonable and appropriate accommodations to examinees with documented disabilities who demonstrate a need for accommodations in accord with the Americans with Disabilities Act. The letter and spirit of the Americans with Disabilities Act is to provide accommodations to ensure a level playing field for all examinees. The few requests that are not made in good faith hurt all examinees because, if such requests are in fact granted, they give those examinees an unfair advantage vis-à-vis all other examinees. Together with jurisdictions, we strive to deliver and administer a secure test under standardized conditions, resulting in exam scores that can be relied upon and an admissions process that is fair.

Until the next issue,

Judy