The Testing Column: Measuring Competence: Assessment of Knowledge and Skills on the Bar Exam

This article originally appeared in The Bar Examiner print edition, Fall 2023 (Vol. 92, No. 3), pp. 21–28.By Danette Waller McKinley, PhD, and Beth E. Donahue

The current Uniform Bar Examination, rooted in the launch of the Multistate Bar Examination (MBE) over a half century ago, is used in 41 jurisdictions to determine whether those seeking to enter the practice of law possess the minimum level of competence to serve the public. Although the examination is regularly reviewed to ensure its link to the profession, the bar exam still faces assertions that it has minimal relation to the practice of law and does not adequately measure skills. This article reacquaints readers with the methods NCBE has used to corroborate the bar exam’s relevance to the practice of law and ensure it continues to test the skills entry-level lawyers need.

The Bar Examination and Relevance to the Practice of Law

Licensure Testing and the Standards for Educational and Psychological Testing

Two of the most important aspects of any testing process are determining what should be tested and collecting evidence that supports the interpretations made about test results. These aspects are referred to as validity evidence.

Guidelines developed jointly by the American Educational Research Association, the American Psychological Association, and the National Council on Measurement in Education reflect best practices in testing and measurement. These guidelines, the Standards for Educational and Psychological Testing, detail practices undertaken by educational, psychological, employment, and credentialing (professional licensure and certification) testing organizations for test development, administration, scoring, and reporting, among other aspects of the testing process.1

The Standards also provide a definition of licensure testing’s purpose, which is to ensure that those who pass have sufficient knowledge and skills to safely practice. The goal is to allow examinees the opportunity to show that they possess these knowledge and skills for general, entry-level, unsupervised practice in a profession.

Conducting Practice Analyses

As an initial matter, looking at what newly licensed practitioners do must be the basis for identifying the necessary knowledge and skills to begin practice. This is often done through a practice analysis or similar study in which new practitioners (and sometimes their supervisors) are asked about the importance and frequency of tasks they perform in practice.

Attorney practice analyses have been conducted by multiple organizations, including NCBE. NCBE conducted its most recent analysis in 2019 and had previously done one in 2012; prior to that, it had conducted content validity studies for the MBE in 1992 and 1982.2 The Institute for the Advancement of the American Legal System (IAALS) conducted its own analyses in 2014 and 2020.3 The State Bar of California (California Attorney Practice Analysis; CAPA) and the Florida Board of Bar Examiners (Florida Practice Analysis) also conducted studies in 2020.4

IAALS’s Foundations for Practice Study

Based on survey results from employers as part of IAALS’s Foundations for Practice project, IAALS identified five skill categories that are important in attorney practice: the attorney as communicator, the attorney as practitioner, the attorney as professional, the attorney as problem solver, and the attorney as self-starter.5 IAALS did additional work to further define the competencies associated with early practice. These competencies, described in IAALS’s Building a Better Bar, are provided in Figure 1.

Figure 1: Lawyer competencies associated with early practice

Source: Institute for the Advancement of the American Legal System, AccessLex Institute, and The Ohio State University Moritz College of Law, “Building a Better Bar: The Twelve Building Blocks of Minimum Competence” (December 2020), available at iaals.du.edu/projects/building-better-bar-capturing-minimum-competence.

The NCBE and California Practice Analyses

The practice analyses conducted by NCBE and the State Bar of California sought to identify what knowledge and skills were needed for entry-level practice. Both surveys produced similar results, although CAPA surveyed California-based practitioners, and NCBE’s survey included a national sample.

Figure 2 compares the results from the two surveys pertaining to foundational knowledge. The difference between the two is that the NCBE results include Business Associations and the CAPA results include Administrative Law and Procedure. However, we should note that there may be areas of overlap even between the subject areas that appear to be distinct. For example, in NCBE’s detailed content scope outlines, some aspects of Administrative Law and Procedure are incorporated into the Constitutional Law outline and some agency-­related topics overlap between Torts and Business Associations.6

In addition to these doctrinal subject areas of foundational knowledge, both analyses identified the skills needed for entry-level practice. Again, there is substantial overlap between the results of the analyses, as shown in Figure 3. NCBE’s list isolates Legal Research in its own category and includes Analysis and Evaluation skills as skills that are related to Issue Spotting and Investigation, whereas CAPA’s list combines Investigation with Research, does not specifically call out Analysis or Evaluation, and includes Litigation as a skill category.

When skills and tasks to be assessed are grounded in a practice analysis, the analysis provides evidence that the test content is related to practice in the context of the test purpose—to ensure that those who pass have the necessary knowledge and skills for general, entry-level, unsupervised practice.

Figure 2: Comparison of necessary foundational knowledge identified in NCBE and CAPA practice analysis results

Figure 3: Comparison of skills needed for entry-level practice identified in NCBE and CAPA practice analysis results

The Bar Examination and the Measurement of Knowledge and Skills

The Standards remind us that there may be several aspects of performance that relate to professional success, but not all of them are directly relevant to licensure testing. For example, a lawyer’s ability to build an extensive professional network or deep ties to a particular community may contribute to success in skills related to time, stress, and practice management, but such aspects of those skills would be difficult to assess fairly in a typical licensure exam.

Similarly, practical constraints such as the number of examinees to be assessed and the costs related to developing different types of questions (costs that may be passed on to examinees) may make it unrealistic to measure some aspects of skills that are best observed in one-on-one or small-group supervised training experiences in school or in early practice.

In short, a licensure exam is not designed to measure all relevant aspects of important knowledge and skills, but rather to assess a fair and realistic sampling of relevant aspects of important knowledge and skills.

All NCBE Exam Questions Integrate Both Knowledge and Skills

In the previous section, we explained how relevant knowledge and skills for the practice of law were identified through practice analysis results used to determine the most important and most frequently reported tasks. Those tasks were then used to identify the skills and knowledge to be assessed on the current and NextGen bar exams.

The bar examination has always measured both skills and knowledge. Indeed, almost any licensure test question measures both a skill or skills and some amount of knowledge. NCBE’s exams have never used pure knowledge or “recall” questions that require only the regurgitation of memorized knowledge. An example of such a recall question, which is not the type of question that would appear on the bar exam, is shown as follows:

Recall Example

Which of the following accurately states the amount-in-controversy requirement for diversity jurisdiction in federal court?

(A) The matter in controversy must be equal to or greater than $75,000, including interest and costs.

(B) The matter in controversy must be equal to or greater than $75,000, exclusive of interest and costs.

(C) The matter in controversy must be greater than $75,000, including interest and costs.

(D) The matter in controversy must be greater than $75,000, exclusive of interest and costs.

Such recall questions (which would not appear, as written, on the MBE or on the NextGen exam) require no lawyering skill or ability; any person, whether trained in law or not, could be expected to answer correctly if they had memorized the applicable section of the United States Code related to the amount-in-controversy requirement. Contrast this recall example with a question in the style of the MBE:

MBE Example

A construction worker sued an insulation manufacturer in federal court, claiming that he had developed a chronic health condition as a result of 20 years of exposure to the manufacturer’s insulation at his work sites. The manufacturer answered, denying all liability and stating that it had never supplied its insulation to the worker’s employer.

The worker’s attorney deposed the manufacturer’s president, and the manufacturer’s attorney deposed the worker. Immediately thereafter, the manufacturer moved for summary judgment on the ground that the worker had no evidence showing that the insulation had ever been used by the worker’s employer.

What would be the worker’s best response to the motion for ­summary judgment? Select one. 

(A) Argue that more time is needed for additional discovery to show the manufacturer’s liability, and attach a declaration describing the desired discovery.

(B) Argue that the motion should be denied, because a central issue in the case will be the manufacturer’s credibility on the question of its distribution of the insulation, and only a jury can decide questions of credibility.

(C) Argue that the motion should be denied, because the manufacturer failed to attach any evidence to its motion to show that the insulation was not used by the worker’s employer.

(D) Make a cross-motion for summary judgment arguing that the manufacturer has introduced no evidence to show that its insulation did not harm the worker.

The use of factual vignettes in NCBE’s exams and other licensing exams creates the opportunity to integrate knowledge with professional skill (in this case, identifying the best response to an opposing party’s argument). Such questions require more than mere regurgitation of knowledge of a relevant rule; they require application of knowledge (such as knowledge of procedural rules of discovery) and a cognitive skill or skills (such as issue spotting and analysis) to a realistic scenario. One would have a hard time imagining a person who has no legal training answering such questions correctly even if they had been given an opportunity to study the relevant rules.

In addition to questions like the above MBE-style example, the NextGen exam will include questions like the following:

NextGen Example

You are representing a client in a dispute with a neighbor. The client owns a single-family home with several acres of surrounding land. Recently, the client noticed that his neighbor had built a fence that extends onto the client’s land. The client is unsure when the fence was built because that part of the client’s land is obscured by large trees. When the client contacted the neighbor about the fence, the neighbor claimed that she did not know the location of the property line. The client has shown you a recent survey of the property that confirms the encroachment and has asked you whether he has a claim against the neighbor.

Which of the following legal topics are the most important for you to research before advising the ­client? Select two.

(A) Adverse possession.

(B) Conversion.

(C) Implied easement.

(D) License.

(E) Negligence.

(F) Trespass.

Like the previous question, this question is based on a factual vignette and requires the application of knowledge and professional skill (not the mere regurgitation of a memorized principle) to a realistic scenario.

On the “skills” side of NCBE’s exams, a similar dynamic is observable: pure skills questions have never been used. But unlike our recall example above, it is difficult to even imagine what a pure-­legal-skill question, devoid of any required knowledge, might look like. The Multistate Performance Test (MPT) is widely viewed as NCBE’s “skills test.” This exam provides examinees with a library of legal resources, such as statutes and judicial opinions, and a multipage client case file, including documents such as transcripts of interviews, police reports, and other documents attorneys regularly work with as part of a legal representation. The exam requires examinees to draft extended legal analyses that apply the provided law to the facts derived from the case files.

While the MPT clearly assesses skill at formulating an original legal analysis based on provided resources, this task necessarily requires legal knowledge: knowledge of the relative weight and authority properly ascribed to varying sources, knowledge of the standard components of legal analysis and how to arrange them effectively, knowledge of the different purposes of common legal documents and the expected tone and tenor of each, and so on.

But if all bar exam questions have always required a mix of knowledge and skill, why is there a pervasive understanding of MBE (and essay) items as being “knowledge” questions and MPT items as being “skills” questions, and what is meant by reports that the NextGen bar exam should be more skills-­focused than the current bar exam?7

On the knowledge side, the answer is relatively straightforward for those familiar with modern bar preparation: when we speak of “knowledge” on the bar exam, “doctrinal knowledge” is typically meant. MBE and essay items typically require knowledge of detailed rules of legal doctrine and how those rules function together within a larger set of rules—not a kind of generalized, “I recall learning it in my first year of law school” knowledge, but memorized, at-the-ready details of how a particular rule operates such that one can apply the rule to any factual vignette presented on the exam in response to an issue-­spotting, analysis, or client-advising prompt.

It is critical to state that law professors, practitioners, and judges put exam questions through extensive review for appropriateness to beginning practice. These review groups are empowered to (and do) reject questions as being too “nitpicky” or “in the weeds” for inclusion on the bar exam.

Even after passing through review groups, questions are pretested before they are used as scored questions on real exams, and they are rejected if they appear to be too difficult or too easy to meaningfully contribute to jurisdictions’ pass/fail decisions. Question formats, drafting techniques, grading practices, and jurisdiction passing scores (the designated pass/fail points on the exam’s score scale) all contribute to controlling the extent of doctrinal knowledge required to perform well. Yet within these carefully constructed constraints, performance on the MBE and MEE remains dependent on a large amount of at-the-ready doctrinal knowledge.

Contrast this requirement of doctrinal knowledge to NCBE’s “skills” test, the MPT. Although it certainly requires the application of lawyering knowledge, it is not doctrinal knowledge but interdisciplinary, practical lawyering knowledge that must be brought to the exam. Demonstration of this kind of knowledge requires working with doctrinal knowledge—but the doctrine is provided, not memorized prior to the exam. Hence, we can imprecisely understand the MBE and MEE to be more “knowledge-­focused” than the MPT, with the clarification that “knowledge” means “doctrinal knowledge.”

On the skills side, the answer is perhaps more subtle, even to those knowledgeable about bar preparation. The MBE-style question shown above is drafted in third-person voice. The facts are provided in prose, briefly describing a legal issue between two parties and asking for the best “next step” for one of the parties to take. In this question, attorneys appear in the facts on behalf of the parties, although in many MBE and MEE questions there is no mention of attorneys. No motives or internal thoughts of the parties are provided in this example, but in some MBE and MEE questions, the narration of facts is omniscient, providing such details to aid examinees in their analyses.

Contrast this factual input from an MBE or MEE vignette to the case file provided in an MPT. There is an introductory memo providing instructions to the examinee in second-­person voice, asking the examinee to perform a task in the role of an entry-level attorney. There is a client that the examinee is representing or a public service that the examinee is fulfilling in performing the task. The facts of the matter must be pieced together from documents describing or transcribing conversations with witnesses, parties, law enforcement, or others. Sometimes the documents include factual ambiguities or contradictions. Although the task required of an examinee taking an MPT involves legal analysis and a conclusion, similar to an MBE or MEE question, the framing of inputs intentionally increases the skill involved to perform well. In cognitive science terms, we have moved from the Miller’s Pyramid level “knows how” in the MBE and MEE to the level “shows how” in the MPT (see Figure 4).8

Hence, while cognitive science models are often messy and imprecise, we share a general understanding that the MPT is more “skills-­focused” than the MBE and MEE, with an asterisk needed to clarify that “cognitive science models may differ.”

Figure 4: Miller’s Pyramid

Source: G.E. Miller, “The Assessment of Clinical Skills/Competence/Performance,” 65(9) Academic Medicine S63–67 (September 1990).

Contact us to request a pdf file of the original article as it appeared in the print edition.