How Are Questions Written for NCBE’s Exams? Part Two: Two Written-Component Question Drafters Share the Process

This article originally appeared in The Bar Examiner print edition, Winter 2019-2020 (Vol. 88, No. 4), pp 22–28.

In this issue, we bring you the second part of a section on how questions are drafted for NCBE’s exams—a process that involves multiple stages of drafting, review, and revision by the volunteer members of NCBE’s 10 drafting committees, with the support of NCBE’s attorney editors.

Part One in the Fall 2019 issue shared the process for developing questions for the Multistate Bar Examination (MBE) and the Multistate Professional Responsibility Examination (MPRE) from members of those drafting committees, including the important step of pretesting that all questions for NCBE’s four exams undergo.

Continue reading to learn from two members of the Multistate Essay Examination (MEE) and Multistate Performance Test (MPT) Drafting Committees about how questions are written for those two exams.

The Gestation of an MEE Question: A Rigorous Process

By Sheldon F. Kurtz

As the longtime chair of NCBE’s Multistate Essay Examination (MEE) Drafting Committee, I ­welcome the opportunity to reflect on the MEE from a drafter’s perspective and to explain the rigorous process by which an MEE question is created.

MEE questions are drafted collaboratively by the MEE Drafting Committee, currently consisting of six people: one appellate judge and five law school faculty members. Over the course of the exam’s existence, the subject matter tested by the MEE has changed. Currently, there are 12 areas of law that may be covered by the questions on any MEE. Generally, only six of these subjects are tested in a particular administration, although some questions may require analysis of more than one subject area. Each of these subject areas is an area of expertise of at least one member of the committee.

Development of an MEE question from initial drafting to test administration typically takes about three years because of the care and attention to detail paid to the development of each question.

For each question on an administered exam, there is at minimum (1) the initial draft; (2) drafting committee review; (3) outside expert review, followed by drafting committee review; (4) pretesting, followed by drafting committee review; (5) assignment to a test form or to another category (test form referring to a particular set of test questions administered at a given time), followed in either case by additional reviews at later meetings; and (6) policy committee review, followed by drafting committee review. Throughout, NCBE staff members, including its attorney test editors, provide another valuable layer of editorial review, which only serves to strengthen the quality of the question.

The First Draft and Drafting Committee Review

After the need for a question in a specific subject area and the specific topic(s) within that area is determined (e.g., from the MEE subject-matter outline, this might be “business associations—corporations and limited liability companies”), a first draft of a question is prepared by either a drafting committee member or an outside question writer (a law school professor). Questions (and analyses, discussed later) are prepared according to NCBE’s MEE Item Writer’s Guidelines, which detail such things as length, style conventions, analysis content and format, and suggested grading percentages. In no case has the final draft of an administered question ever looked like the first draft, because each question goes through multiple stages of review and revision.

When a newly commissioned question is received, it is assigned to a “shepherd,” a drafting committee member (who is not the author of the question and who is a content expert in the question’s subject area). The shepherd is responsible for leading the discussion and editing of the question through the first meeting at which it is reviewed by the entire drafting committee. At that first meeting, the entire committee determines whether the question is appropriate for a future test. If so, the committee offers some initial editorial suggestions for the shepherd to review outside the meeting. In addition, the committee decides whether to collectively review the question again or, following the shepherd’s edits, to move it forward to one or both of the next test development steps.

Outside Expert Review and Pretesting

Once the drafting committee determines that a question is ready to move forward, it is sent for outside expert review to a law professor or an established practitioner and is also pretested by newly licensed lawyers. The reviewers are given guidelines to follow in preparing their reviews, including questions to answer about the analysis.

Pretesters are identified and invited by bar administrators across the country to take the MEE under exam-like conditions using an application that records how much time is spent on each question. They complete a survey about each question, rating it on clarity of facts and call of the question, difficulty, bias, and appropriateness in general. They also participate in a debriefing session with the bar administrator from their jurisdiction, who then prepares a memo that the drafting committee reviews at its next meeting.

The drafting committee carefully considers the results of the outside expert review and pretesting, which always lead to significant improvements to the question. The committee then either assigns the question to a future exam form or places it in a test-ready category for eventual assignment to a test form.

Further Review After Assignment to a Test Form

Because a test form is typically assembled 18 months before its administration, the drafting committee will typically review a question at least two more times following the question’s assignment to a test form. Then, a few months before the scheduled test’s administration, the test form with all six questions is sent to the MEE/Multistate Performance Test (MPT) Policy Committee for review and comment. The drafting committee may make further changes to the questions based on these comments.

By now, it should be clear that questions are not simply drafted shortly before a test administration. It’s a much longer process with numerous quality-control steps built into the making of the final product.

Developing the Summary and Analysis

For each MEE question, the drafting committee also drafts a summary and an analysis. The analysis is provided to the jurisdictions to assist graders in grading the MEE; it is illustrative of the discussion that might appear in an excellent answer to the question. As much care goes into the drafting of the summary and analysis as goes into the drafting of the question.

The summary is a shortened version of the analysis and represents the drafting committee’s judgment of what a high-quality answer could be. The analysis, however, includes much more information than the summary because it has the additional goal of educating the graders assigned at the jurisdiction level about the subject matter being tested. Furthermore, the analysis provides graders with a detailed discussion of the issues to help them appropriately rank-order answers and often makes suggestions to graders as to what discussion in an actual answer is worthy of credit.

Preparation of the summary and analysis is an absolutely essential part of quality control. It ensures that a question’s fact pattern fully supports the conclusion and rationale reflected in the summary and analysis. And if, in the early stages of test development, the question and the summary/analysis are out of sync, it signals that changes in the facts or analysis must be made.

It is important that the graders fully understand the facts and analysis of the question so that they are able to grade essay answers in a consistent manner. Accordingly, following each MEE/MPT administration, NCBE holds a Grading Workshop to support jurisdiction graders in their grading of the MEE and the MPT; at these workshops, drafting committee members act as facilitators during hands-on grading sessions.

The Fruits of a Hardworking Drafting Committee’s Labor

Perhaps the most impressive thing I’ve seen over my years on this drafting committee is the intensity and devotion with which committee members approach their work and the pride that they take in ensuring that MEE questions are of the highest possible quality. The time commitment of the faculty members, judges, and lawyers serving on the committee is enormous. In addition to the time throughout the year that each committee member spends drafting and editing questions (which itself is substantial), the committee meets twice a year for two consecutive 8- to 9-hour days.

Furthermore, the committee’s commitment to quality necessitates an enormous amount of wordsmithing to the question, the summary, and the analysis. I’m very proud to say that this process (as tedious as it sometimes feels during a meeting) has paid off.

At the core of every successful project are people. It is the dedicated committee members and NCBE staff with whom I am honored to work who make the MEE test development process a success.

Drafting MPT Items: Guiding Principles and Collaboration

By Alexander W. Scherr

Drafting items (or questions) for the Multistate Performance Test (MPT) requires a special set of abilities: knowledge of the law, careful writing, imagination and creativity, and good judgment. The MPT Drafting Committee includes law professors and lawyers from diverse practices who have worked with young lawyers and law students. The drafting process for an MPT item takes about two years and a high degree of collaboration. This article describes the format and the drafting process for an MPT item. It ends by articulating some guiding principles for and reflections on the work of drafting a good item.

The Format of an MPT Item

An MPT item asks examinees to complete a legal task in writing based on a universe of facts and law included in the MPT materials. Each MPT item has two parts: a File and a Library.

The File starts with a memo that contains instructions to the examinee from an assigning attorney, often followed by directions on how the examinee should format the written product. The balance of the File contains facts presented in diverse formats, including transcripts of hearings or interviews, results of law firm investigations, newspaper articles, or excerpts from contracts, pleadings, or other documents. Drafters avoid maps, graphics, or complex forms, which are difficult to replicate in those versions of the test created for examinees who have received ADA accommodations for visual disabilities.

The Library contains the law necessary to accomplish the task. Libraries often include selected statutes and cases but may also contain regulations, ethical opinions, Restatement provisions, and secondary sources. Drafters use either federal or state sources, as appropriate, making sure to select well-accepted legal principles.

Examinees must use these materials to complete a task that a young lawyer in practice might perform: an objective memo, a persuasive brief, a contract or other transactional document, a proposed judicial order for a judge, a client letter, or a pleading, among others.

All the necessary facts appear in the File; all the necessary law appears in the Library. For this reason, the MPT is not a test of the examinee’s knowledge of substantive law. Instead, it tests the examinee’s ability to read carefully, to sort and organize facts, to identify relevant legal principles, and to integrate facts and law to accomplish the task set in the memo through careful organization and writing.

The Drafting Process

Drafting an MPT item typically takes about two years and includes several stages of drafting, critique, editing, and review, including at least three edits by the drafting committee and critical assessments from a wide range of sources.

The First Draft and First Committee Review

An MPT item starts with an individual drafter on the committee, who identifies an area of the law, a specific scenario within that area, and a task for the examinee to perform. The drafter produces both an initial draft and a point sheet. This point sheet describes the item in detail and outlines the issues and grading points that jurisdictions will eventually use in grading the item.

NCBE staff members also arrange a pretest of the item, with 8–10 newly licensed lawyers selected by bar administrators taking the MPT item under exam-like conditions and providing feedback about the item. The bar administrator also prepares a report summarizing the feedback from the pretesters about the item.NCBE also appoints a separate member of the committee as a “shepherd,” with responsibility to lead discussion and editing of the item. Drafter and shepherd discuss the item early in the process, making sometimes extensive edits to the item before it is sent to the full committee. All committee members review the item and come prepared to comment at the next semiannual drafting meeting. At that meeting, the shepherd presents the draft, identifies significant areas for group discussion, and leads a line-by-line review of the item.

Outside Expert Review, Pretesting, and Policy Committee Review

After revisions in the first meeting, NCBE staff members submit the item to outside expert reviewers: one law professor and one practitioner with expertise in the pertinent area of law. NCBE staff members also arrange a pretest of the item, with 8–10 newly licensed lawyers selected by bar administrators taking the MPT item under exam-like conditions and providing feedback about the item. The bar administrator also prepares a report summarizing the feedback from the pretesters about the item. Experts and pretesters sign confidentiality forms to prevent disclosure of the item or its topic.

NCBE staff members send the pretest answers, the pretest report, and the expert reviews to the drafting committee, which reviews these materials and grades the pretest answers before the next meeting. At that meeting, a new shepherd prepares a list of edits suggested by these materials, which the committee incorporates as appropriate.

When items are assigned to a test form, they receive another review by NCBE’s Multistate Essay Examination (MEE)/MPT Policy Committee, consisting of bar examiners from MEE and/or MPT user jurisdictions. The policy committee comments come back to the drafting committee for a third review of the items and further editing before the items are finalized.

Developing the Grading Materials

Once the third review is complete, NCBE attorney editors take the final steps to prepare the question for the bar exam. They prepare grading materials that include the point sheet, several summaries of grading weights and points, and a model answer. During this process, NCBE attorney editors often work with the drafter or earlier shepherds of the item to finalize these materials, which sometimes produces additional edits before the item appears on an upcoming exam.

Six Guiding Principles for Drafting MPT Items

The following six principles guide the drafters and editors of MPT items.

1. The item should be realistic.

Items should present facts and require the performance of tasks that address problems that a lawyer will likely see in practice. Most items include both relevant and irrelevant facts, testing the examinee’s ability to sort and winnow. The legal sources reflect real-world majority or minority approaches to the scenario posed by the item.

2. The task should pose challenges that occur in real practice.

The task should be one that a newly licensed lawyer would encounter in practice. The practice contexts for items may vary, including work in private firm, nonprofit, governmental, and judicial practices. But the item’s subject matter and its assigned task should offer a fair test of examinees’ ability to perform work they will see early in their careers.

3. The item should require answers that cover a range of difficulty.

Each item typically includes several different grading points, which should range in difficulty from easier to more difficult. Almost every item has one or two relatively easy points as well as points that only better examinees will see and address. This practice should lead to answers that allow graders to rank-order examinees reliably by the quality of their performance.

4. Items should cover a variety of tasks.

Over time, the items presented on exams should present tasks drawn from a wide range of practice areas. Items should not focus solely on writing memos, but should call for work characteristic of litigation, transactional, and judicial practices. NCBE’s MEE/MPT Program Director monitors the bank of completed items and encourages drafters to create items that fill in any gaps in coverage.

5. The item should fairly reflect skills gained during law school.

Items should fairly reflect the training that examinees will have received in law school. Assessing the tax consequences of a complex corporate reorganization would, for most law students, fall outside the abilities they develop in law school. By contrast, writing a persuasive argument or assessing a contractual provision would fall within the average law student’s training.

6. An examinee should be able to complete an answer within 90 minutes.

The pretest process helps the committee to determine whether the item can be read and completed within the 90-minute time span of an MPT. Pretesters record how long it takes them to complete an item and provide feedback on the time pressure they experience. This helps the committee assess whether the item needs rewriting to increase its readability and whether the item has too many or too few issues for the examinee to address.

Three Observations about the Drafting Process

Three aspects of the drafting process stand out from my nearly 15 years of drafting MPTs. First, collaboration is central to the process. Initial drafters rely on feedback from the shepherds and the committee that draws on a wide range of knowledge and perspectives; item editors play a critical role as constructive readers. NCBE fosters this collaboration with a web of useful resources: outside experts, pretest participants, bar examiners, copy editors, and experienced NCBE staff members.

Second, because it is a written exam, the MPT cannot test skills that do not involve written performance: interviewing a client, questioning a witness, negotiating a settlement or a contract, or managing the flow of a trial. That said, MPT items serve as useful proxies for testing related skills, including specifically the ability to deal with facts and to solve problems within the limits of the law and the range of client goals and needs.

Finally, MPT items differ importantly from essay and multiple-­choice tests, in that they test problem-solving in a plausible real-world setting. Rather than testing memory or acquired knowledge, MPT items test how well examinees accomplish a set of goals with materials that they first encounter in the exam setting. In so doing, the MPT tests the examinee’s capacity to exercise judgment in serving a client and thus plays a unique role in assessing whether an examinee has the ability necessary for admission to the bar.

Contact us to request a pdf file of the original article as it appeared in the print edition.