This article originally appeared in The Bar Examiner print edition, September 2015 (Vol. 84, No. 3), pp 23–28.
Update: As of February 2017, the MBE consists of 175 scored questions and 25 unscored pretest questions.
By C. Beth HillWhen I joined NCBE as program director for the Multistate Bar Examination (MBE), I quickly learned that for the MBE question-development process, timing is everything. For example, when we send the February 2016 MBE to the printer in the fall of 2015, we will have already printed the July 2016 MBE and will have selected questions (or “items”) for the July 2017 MBE. (As explained later, the July MBE is printed before the February MBE each year.) Long before an examinee is presented with an item in an MBE test booklet, one of the MBE’s seven committees of subject-matter experts has worked to draft and fine-tune the item, and the item has been revised by an NCBE test editor, reviewed by external experts, and proofread by NCBE’s editorial assistant.
Background Information on the MBE
The MBE is a six-hour, 200-question multiple-choice examination developed by NCBE and administered by 54 jurisdictions as part of their bar examinations. The purpose of the MBE is to assess the extent to which an examinee can apply fundamental legal principles and legal reasoning to analyze given fact patterns.
Of the 200 multiple-choice questions in any given administration, currently 190 are scored questions and 10 are unscored pretest questions (questions whose performance will be evaluated for use on a future exam). The pretest questions are indistinguishable from those that are scored, so examinees are advised to carefully answer all questions. The 190 scored questions on the MBE are distributed among the following seven subjects (there are 27 questions for each subject, with the exception of 28 questions for Contracts): Civil Procedure, Constitutional Law, Contracts, Criminal Law and Procedure, Evidence, Real Property, and Torts. (A detailed subject matter outline listing the MBE’s potential scope of coverage on the test is available on the NCBE website.)
The MBE is administered on the last Wednesday in February and the last Wednesday in July of each year. The exam is divided into morning and afternoon testing sessions of three hours each, with 100 questions in each session. The MBE is combined with the Multistate Essay Examination (MEE) and the Multistate Performance Test (MPT) to form the Uniform Bar Examination (UBE), which has been adopted in 18 jurisdictions. (This includes Iowa and Kansas, with their first administrations of the UBE in February 2016, and New York and Vermont, with their first UBE administrations in July 2016.)
MBE Drafting Committees
MBE questions are developed by seven MBE drafting committees, one for each subject covered on the MBE. The drafting committees, each with five to seven members, are composed of a mix of the following experts: law professors, practicing attorneys, and judges. MBE drafters are experts in their respective subject-matter fields. Each MBE drafting committee meets twice a year to review and edit questions over the course of a long weekend after the initial drafting of the questions has occurred. Members presently serving on the MBE drafting committees hail from 22 jurisdictions and represent 22 law schools as well as federal courts, government offices, and law firms.
Four test editors—all lawyers—provide direct support to these committees. Additionally, many NCBE staff members provide administrative, technical, and editorial support for the MBE.
Developing the MBE: The Basic Steps
Drafting and Editing
MBE questions are written by MBE drafting committee members. The items are written in accordance with NCBE’s MBE item-writing guidelines, which are based upon best practices from psychometric research and detailed style conventions. Each item can be broken down into three components: the factual scenario, the call of the question, and the “options” or answers. For each semiannual round of drafting, the drafters are assigned specific areas within the MBE subject matter outline (for example, from the MBE subject matter outline for Contracts, one assigned area might be Remedies—Rescission and Reformation). Authors are reminded that the exam is designed to test minimum competence of a newly licensed lawyer, so the question must reflect that level of difficulty within the subject area. After an item is submitted to NCBE, a staff member enters it into NCBE’s question-management database. The database allows staff to track edits, make notes, and chart the item’s life cycle from development through exam performance.
Once the item is entered into the database, an NCBE test editor reviews the item, inserts relevant queries about the scenario, and suggests edits based on the item-writing guidelines. Because there are several drafting committees, the NCBE test editors often collaborate with one another during the initial editing phase to ensure that the same editorial rules are applied, regardless of subject area. After an item has been edited, either the drafter or the committee chair reviews the suggested edits and decides which edits to accept. Then the item is ready for the full committee to review at its next meeting.
The semiannual committee meetings are the first opportunity for all members of a committee to review and comment on the new items that have been submitted. In preparation for the meeting, each committee member is tasked with reviewing the new items and developing suggestions for how the items can be improved. The review of the new items during the meeting is a collaborative effort, with input from each member encouraged. After the committee members agree on any needed improvements, an item is promoted to “exam ready” status. Occasionally, a committee will decide that an item will not likely work well (for example, if the item is delving into an area of law that is too nuanced for a test of minimum competence). In those cases, the item is either retired (never placed on an exam) or assigned to a committee member to rewrite for another review at a future meeting.
Outside Expert Review
Once an MBE item has undergone thorough editing by its drafting committee and test editor, it is subject to additional review by outside content experts. The outside experts, one practitioner and one law professor for each subject area, assess whether the question tests a core concept (essential to the base of knowledge and skills that a newly licensed lawyer is expected to possess), is set in a realistic fact scenario, and is free of bias (gender, racial, regional, or other). The reviewers provide written feedback for the item, and any concerns are returned to the drafting committee for consideration.
Pretesting
As mentioned, of the 200 multiple-choice questions in a set of AM and PM MBE booklets, 190 are scored questions and 10 are unscored pretest questions that are indistinguishable from those that are scored. For each test administration there are multiple versions of the AM and PM MBE booklets. While each version contains the same scored items, different versions have different sets of pretest questions. Pretesting is an important component of building a reliable examination because it allows items to be tried out to evaluate their quality. In order to be used as a scored item on a future MBE, a pretest item must meet NCBE’s statistical standards. For example, the item cannot be too easy or too difficult, and it must show evidence that examinees who answered it correctly also tended to obtain high scores on the entire exam. Pretest items that do not meet these standards are rewritten by the drafting committee or retired.
Creating MBE Test Booklets
The process for choosing items that will eventually be printed in the test booklets, creating and proofreading the booklets, and sending the booklets to the printer begins long before the administration date. First, NCBE test editors, working with the drafting committee chairs, select items from the exam-ready pool to use as scored items. The items are selected in conformance with the subject matter outline for each content area and statistical criteria. A select number of scored items serve as “equator” items. Equators, drawn from scored items on multiple previous administrations, are used to ensure the comparability of the exam’s scaled scores to the scaled scores earned by examinees in previous MBE administrations. As a set, equators comprise a mini-exam that is representative of the total exam from both content and statistical perspectives. (See this issue’s Testing Column for more information about equators.)
Next, the drafting committees review the proposed set of scored items at their next two semiannual meetings. The purpose of this review is for the committee members to confirm that each item is testing current law and that the answer denoted as correct is in fact the correct answer. If the committee finds that an item needs to be replaced, the NCBE test editors provide replacement options.
After each drafting committee has reviewed a proposed set of scored items twice, the test editors and the committee chairs select the 100 pretest questions that will be distributed among the MBE test booklets. Printing preparation occurs approximately 9 months prior to a February administration and approximately 20 months prior to a July administration. We print July examinations (with larger volumes due to the higher number of July examinees) more than a year in advance, so that in the event of a security breach we are able to maintain the administration schedule by substituting the next July exam for either the current July or upcoming February exam.
Before the exam is sent out to be printed (by an outside printer supervised by NCBE, selected in part for its demonstrated capability for printing confidential materials), NCBE staff members divide the items into the two booklets for the standard testing sessions: morning, or “AM,” and afternoon, or “PM.” Items are assigned to each booklet in a manner that ensures that the known average difficulty of items in each session is comparable and that items are fairly equally distributed by subject area in each session. NCBE staff review item placement for each AM and PM booklet for each version of the exam. Each booklet is also reviewed by NCBE’s editorial assistant before final versions are sent to the printer.
After NCBE staff members approve the printer proofs, each standard version of the MBE is printed. As a final quality control step, the printer ships samples to NCBE, which staff members review, looking for ink splatters or folded pages that could impede an examinee’s progress or concentration. Staff members also take one last look to make sure that every item has four options, that there are a total of 200 items, and that each page is in the correct order. Booklets are stored at the printer while NCBE takes orders for the upcoming administration. To accommodate examinees with disabilities, the following nonstandard materials are produced by NCBE and shipped to jurisdictions that order them: Braille booklets, split booklets (used to separate the exam into four, six, or eight sections when additional time has been granted), large-print booklets (in both 18-point and 24-point fonts and in split sets), audio CDs, and files for laptops (owned by NCBE) equipped with specialized assistive-technology software.
After the jurisdictions have placed their orders, the standard booklets are packed and shipped to jurisdictions by the printer, and the nonstandard materials are shipped directly from NCBE. Because it is vitally important that the booklets remain confidential, each shipment contains security reminders, and NCBE provides jurisdictions with detailed information on how to respond in the event of a security breach.
Exam Administration and Beyond
The MBE is administered locally by 54 jurisdictions in accordance with strict guidelines provided by NCBE. The guidelines range from what time examinees can open the AM and PM booklets, to how the booklets should be collected and shipped back to the printer (the printer accounts for every single booklet that was shipped out). After the MBE is administered but before scaled scores are calculated, NCBE staff members conduct a preliminary analysis to evaluate the statistical performance of each of the 190 scored items on the exam. Items needing further screening are reviewed via teleconference by drafting committee members. (Further screening may be warranted if, for example, an item is extremely difficult, or if examinee comments that were reported either to jurisdiction staff on the day of the exam or directly to NCBE indicate the need for such screening.) The subject-matter experts then discuss whether there are any potential errors or whether examinees might have been misled by something in the item, and confirm that the correct answer is accurately designated in the scoring system.
When the preliminary analysis is complete and NCBE staff have verified that all answer sheets have been received, NCBE psychometric staff begin the task of equating the MBE to calculate scaled scores. Equating adjusts for variations in test difficulty compared to previous administrations, which ensures that examinees are not unfairly penalized or rewarded for receiving a slightly more or less difficult set of questions. Equating also ensures that the MBE scaled scores are comparable over time. (See this issue’s Testing Column for more information about equating.) After equating is complete, examinees are assigned scaled scores based on the number of questions answered correctly. Then, at about the same time scaled scores are released, printing preparation for an upcoming administration begins again.
As program director for the MBE, I would like to express my appreciation for the dedication of the drafting committee members, who work on this high-stakes licensure exam mostly in anonymity to protect test security. Their commitment to crafting quality items is demonstrated by their attention to detail, and their content knowledge is irreplaceable. I would also like to thank all of the external reviewers, who provide assurance that the items are appropriate for testing the knowledge and legal reasoning skills that a newly licensed lawyer should possess. Finally, I would like to thank my coworkers at NCBE and the jurisdiction administrators and proctors who, as a team, help to ensure that the MBE booklets are properly produced, distributed, administered, and scored.
Sample MBE Questions
Civil Procedure Question
A woman from State A filed an action against a retailer in a state court in State B. The complaint alleged that the retailer had not delivered $100,000 worth of goods for which the woman had paid.
Twenty days after being served, the retailer, which is incorporated in State C and has its principal place of business in State B, filed a notice of removal in a federal district court in State B.
Was the action properly removed?
(A) No, because the notice of removal was not timely filed.
(B) No, because the retailer is a citizen of State B.
(C) Yes, because the parties are citizens of different states and more than $75,000 is in controversy.
(D) Yes, because the retailer is a citizen of both State B and State C.
Answer: B
Constitutional Law Question
A state law prohibits any barbershop licensed by the state from displaying posters in support of any current candidate for public office or displaying or distributing any campaign literature in support of such a candidate. No other kinds of posters or literature are subject to this prohibition, nor are any other types of commercial establishments in the state subject to similar prohibitions.
Is this law constitutional?
(A) No, because it treats barbershops differently from other commercial establishments.
(B) No, because it imposes a restriction on the content or subject matter of speech in the absence of any evidence that such a restriction is necessary to serve a compelling state interest.
(C) Yes, because it leaves political candidates free to communicate their campaign messages to voters by other means.
(D) Yes, because the operation of a licensed barbershop is a privilege and, therefore, is subject to any reasonable restriction imposed by the state.
Answer: B
Contracts Question
A company contracted with a builder to construct a new corporate headquarters for a fixed price of $100 million. At the time of the contract, structural steel was widely available and was included in the contract as a $6 million item. Before work began on the project, tornado damage shut down the production facility of the biggest structural steel supplier in the country, and the price of structural steel increased by 20% as a result. The builder informed the company of the steel price increase, and the parties then orally agreed to increase the project price to $101 million.
The builder proceeded with construction and delivered the project on time. The company paid the builder $100 million but refused to pay the additional $1 million.
If the builder sues the company for $1 million, is the builder likely to prevail?
(A) No, because the modification was never reduced to a writing signed by the party to be charged.
(B) No, because there was no consideration for the modification of the contract.
(C) Yes, because the company’s promise was supported by consideration.
(D) Yes, because the modification was fair and equitable in view of the unanticipated increase in the price of structural steel.
Answer: D
Criminal Law and Procedure Question
The president of a pharmaceutical firm received a report from his testing bureau that a manufactured lot of the firm’s anti-cancer prescription medication was well below strength. Concerned about being able to fulfill contractual commitments, the president instructed his staff to deliver the defective lot. A cancer patient who had been maintained on the drug died shortly after beginning to take the defective pills. Medical evidence established that the patient would have lived longer had the drug been at full strength, but would have died before long in any event.
The president was convicted of murder. On appeal, he argues that his conviction should be reversed.
Should the conviction be reversed?
(A) No, because the intentional delivery of adulterated or mislabeled drugs gives rise to strict criminal liability.
(B) No, because the jury could have found that the president’s conduct was sufficiently reckless to constitute murder.
(C) Yes, because distribution of the defective lot was only a regulatory offense.
(D) Yes, because the cancer, not the president’s conduct, was the proximate cause of death of the patient.
Answer: B
Evidence Question
A man sued a railroad for personal injuries suffered when his car was struck by a train at an unguarded crossing. A major issue is whether the train sounded its whistle before arriving at the crossing. The railroad has offered the testimony of a resident who has lived near the crossing for 15 years. Although she was not present on the occasion in question, she will testify that, whenever she is home, the train always sounds its whistle before arriving at the crossing.
Is the resident’s testimony admissible?
(A) No, due to the resident’s lack of personal knowledge regarding the incident in question.
(B) No, because habit evidence is limited to the conduct of persons, not businesses.
(C) Yes, as evidence of a routine practice.
(D) Yes, as a summary of her present sense impressions.
Answer: C
Real Property Question
A brother and a sister purchased land under a deed that conveyed title to them as joint tenants with right of survivorship. Common law joint tenancy is unmodified by statute in the jurisdiction.
The purchase price was $50,000, of which the sister paid $10,000 and the brother paid $40,000. The sister later mortgaged her interest in the land. The brother then died testate, leaving his entire estate to a cousin. The sister later paid off her mortgage debt, and the mortgage was released.
At the present time, who owns the land?
(A) The answer depends on whether the jurisdiction follows the lien theory or the title theory of mortgages.
(B) Title is entirely in the sister as the surviving joint tenant.
(C) Title is in the sister and the cousin as equal tenants in common.
(D) Title is in the sister and the cousin as tenants in common, with the sister having a 20% interest and the cousin having an 80% interest.
Answer: A
Torts Question
A man has four German shepherd dogs that he has trained for guard duty and that he holds for breeding purposes. The man has “Beware of Dogs” signs clearly posted around a fenced-in yard where he keeps the dogs. The man’s next-door neighbor frequently walks past the man’s house and knows about the dogs’ ferocity. One summer day, the neighbor entered the man’s fenced-in yard to retrieve a snow shovel that the man had borrowed during the past winter. The neighbor was attacked by one of the dogs and was severely injured.
In a suit against the man, is the neighbor likely to prevail?
(A) No, because the neighbor knew that the man had dangerous dogs in the yard.
(B) No, because the neighbor was trespassing when he entered the man’s property.
(C) Yes, because the neighbor was an invitee for the purpose of retrieving the shovel.
(D) Yes, because the man was engaged in an abnormally dangerous activity.
Answer: A
C. Beth Hill is the Program Director for the Multistate Bar Examination for the National Conference of Bar Examiners. Hill received her J.D. from the Hamline University School of Law and holds a Master of Public Policy degree from the University of Southern California.
Contact us to request a pdf file of the original article as it appeared in the print edition.
