Letter from the Chair

This article originally appeared in The Bar Examiner print edition, Winter 2017-2018 (Vol. 86, No. 4), pp 1–3.

By Hon. Rebecca White Berch

High-stakes tests such as the bar exam have frequently been the subject of criticism. They are often said to be both over- and under-inclusive; that is, they test too much in some areas and too little—or not at all—in others. Some claim that such tests don’t assess the right things; others, that they fail to test skills thought to be necessary for success in the given field. In the case of the bar exam, such untested skills include researching, interviewing, counseling, and writing—except insofar as basic writing skills are used to convey thoughts on the essay and performance test portions of most bar exams. And the bar exam certainly doesn’t test characteristics thought to be necessary for success in practice: grit, courtesy, and compassion, and the abilities to get along, work in groups, and timely return phone calls and emails.

As for those taking high-stakes tests, some seemingly bright and prepared examinees claim that they just don’t test well, so in their view, high-stakes tests are unfair. And, of course, even bright, well-prepared examinees can find themselves having a bad day on exam day because of illness or other stressors.

Given these criticisms, why do jurisdictions persist in giving the bar exam at all, and why does NCBE persist in producing it?

First, the state regulating courts take seriously their obligation to protect the public, and the bar exam provides a piece of that protection by requiring applicants to demonstrate that they possess the minimum knowledge and skills necessary to enter practice. So it is unlikely that jurisdictions will abandon the bar exam requirement.

In times past, the regulating courts from a few jurisdictions relied solely on a student’s completion of a legal education from a law school approved by the ABA as satisfactory proof of the student’s readiness to practice law. (Wisconsin, for example, has long had and still maintains the diploma privilege for graduates of its two law schools.) The ABA recognizes that graduation, even from an ABA-approved law school, may not be enough to protect the public. The preface to the ABA’s Standards and Rules of Procedure for Approval of Law Schools states that “[t]he Council and the ABA believe . . . that every candidate for admission to the bar should have graduated from a law school approved by the ABA and that every candidate for admission should be examined by public authority to determine fitness for admission.”1 The decline in the numbers of applicants to law schools earlier in this decade and the increase in the percentage of applicants with low scores on the Law School Admission Test, coupled with low attrition rates for poor academic performance, further suggest that courts won’t soon forego the bar exam as part of the package of safeguards they require to protect the public.2

Second, there are many things that the bar exam does well. Through a variety of assessment methods, it requires examinees to demonstrate their knowledge of basic legal principles and their possession of basic skills needed for minimum competence to practice law. Multiple-choice questions test examinees’ knowledge of fundamental legal principles and legal reasoning skills and their ability to apply these to fact situations like the ones they might encounter in practice. Essay questions test examinees’ ability to identify the legal issues raised in a question and demonstrate an understanding of the relevant legal principles, while organizing their thoughts and presenting an analysis in clear and concise writing, much as they will be required to do in practice. Performance test questions, for those jurisdictions that administer them, require examinees to complete a task that a new lawyer should be able to accomplish, thereby providing a means by which to evaluate whether the candidate possesses certain fundamental lawyering skills.

The tests developed by NCBE for use by jurisdictions—the Multistate Bar Examination, the Multistate Essay Examination, and the Multistate Performance Test—assess what is needed for entry-level practice. To confirm that this is the case, NCBE commissioned a comprehensive job analysis study, completed in 2012, to determine the job activities performed by new lawyers and the associated knowledge, skills, and abilities (KSAs) required for practice. The results of that study confirmed that the KSAs tested on the bar exam components developed by NCBE are those that new lawyers regularly use and need to possess. Over the next three to five years, NCBE will be supplementing the 2012 job analysis study with additional qualitative validity studies to gather data for the current bar examination and to generate future-focused data to inform what should be tested in the next 7–10 years. These studies will also guide preparations for our next full job analysis study tentatively planned for 2021–2022.

The tests developed by NCBE undergo a rigorous test development process, resulting in high-quality questions that can be relied upon to assess whether candidates are prepared for entry-level practice. Questions are written by drafting committees composed of a mix of law school professors, practicing attorneys, and judges. Each question is carefully edited, reviewed by outside content experts, and then pretested before being used as a live test item. Each question is reviewed after the exam administration to evaluate its performance; in the case of the MBE, an analysis to evaluate the statistical performance of each question is conducted by NCBE’s Testing and Research Department—an outstanding group headed by Dr. Mark Albanese that includes a widely published and highly regarded cast of researchers and psychometricians.

NCBE constantly seeks to improve its products and processes. Through its professional staff and the NCBE Board of Trustees’ Long Range Planning Committee, NCBE regularly monitors changes in the legal profession to determine whether changes to the bar exam are necessary and appropriate. In addition, NCBE staff and policy committees follow relevant research and developments in the field of testing, and stay informed about licensing processes in other professions and in other nations. To continue the quest for improvement, the Board of Trustees has authorized the creation of a Testing Task Force to provide consistent yet dynamic leadership to undertake a future-oriented study of the bar examination. This study will draw in part upon the validity studies discussed above to identify appropriate content and to consider how, if at all, the bar exam should change in terms of format, delivery platform, and grading processes. The Task Force will likely look closely at step testing (such as is done in the medical and accounting professions, where testing occurs as a series of steps to be accomplished during the different phases of a student’s education), computer-based testing, and alternative methods of skills testing, among other issues. The Task Force will work with a technical advisory panel of outside measurement experts from a variety of backgrounds who will provide technical and psychometric guidance. The Task Force will conduct its study during the next three years.

In re-imagining what the bar exam might become, however, we shouldn’t throw out the good in pursuit of the perfect. NCBE understands that the bar exam does not test some skills important to the practice of law. But it is yet to be determined how skills such as interviewing, negotiating, and counseling might be fairly, practically, reliably, and cost-­effectively tested. The current bar exam focuses on testing the basic knowledge, skills, and abilities that the 2012 job analysis study, as well as other sources of information, has shown to be generally agreed upon as being important for new lawyers.

As always, we welcome your comments as we constantly look for ways to improve the process by which candidates for the practice of law are assessed, to ensure that those candidates enter the profession as qualified and practice-ready lawyers.

Sincerely,

Hon. Rebecca White Berch

Contact us to request a pdf file of the original article as it appeared in the print edition.