The Testing Column: Test Validity and the Law: An Overview of Case Law Challenging the Validity of Standardized Tests

This article originally appeared in The Bar Examiner print edition, Summer 2022 (Vol. 91, No. 2), pp. 23–31.By Ken Kraus*

In case law, the validity of standardized tests raises legal issues in at least five areas: (1) equity and fairness, (2) uniform guidelines and test standards, (3) passing standards and scoring challenges, (4) accommodations, and (5) security.

Validity in the psychometric sense means the degree to which a test measures what it claims to measure. The sidebar below, “Validity and the Bar Exam,” discusses NCBE’s commitment to validity in its exams and provides context for the significance of the cases considered in this article. In many of these cases, the plaintiffs are challenging a test’s psychometric validity, including in claims involving discrimination, due process, and accommodations. Plaintiffs are challenging psychometric validity when they claim that test preparers did not use best practices or follow recommended guidelines and standards. Cases involving test security also implicate validity: scores are not valid if the test is not secure. In other cases, plaintiffs claim that a test-related process is not valid in the layperson sense, such as those involving setting the passing score and providing for appeals.

Equity and Fairness

In the area of equity and fairness, litigants have challenged test validity under employment discrimination laws and on due process grounds.

Employment Discrimination Claims

First, litigants rely on Title VII (the Civil Rights Act of 1964),1 passed to outlaw employment discrimination, to challenge the validity of examinations. Under Title VII it is illegal to discriminate based on race, color, religion, sex (including sexual orientation/identity/pregnancy), or national origin.

Under Title VII, plaintiffs can claim disparate treatment or disparate impact. In a disparate treatment claim, the plaintiff must prove intentional discrimination. However, evidence of a discriminatory motive often is not easy to develop. Therefore, plaintiffs generally rely instead on a disparate impact theory of discrimination.

The Burden of Proof

In the key disparate impact case, McDonnell Douglas v. Green,2 the US Supreme Court created an evidentiary framework for analyzing a disparate impact discrimination claim in the context of a summary judgment motion.

As modified by the 1991 amendment to Title VII, plaintiffs must allege that “a particular employment practice … causes a disparate impact on the basis of race, color, sex, or national origin,” which is not “job related for the position in question and consistent with business necessity.”3 For a disparate impact claim, the plaintiff typically relies upon statistical evidence showing that employees or applicants in a protected class are significantly underrepresented for jobs or promotions.

If the plaintiff meets their burden, the burden of proof shifts to the defendant to show that the challenged practice is job-related for the position in question and consistent with business necessity. If the defendant meets their burden, the plaintiff must show that the employer has refused to adopt an “alternative employment practice” with less adverse impact.

Title VII as Applied to Testing

In the context of a challenge to an employer’s use of a test to help the employer make hiring or other employment-related decisions, plaintiffs who meet their initial burden of proof (i.e., showing an adverse impact) must respond to an employer’s showing of a legitimate business interest. They must show that other tests or selection devices would also serve the employer’s legitimate interests with less adverse impact on the protected category of individuals. This is often a difficult challenge for a plaintiff in the context of tests, because alternative tests may not be available or, if available, they might measure different constructs.

Title VII has been found to be inapplicable to exams used for licensing, as opposed to employment, purposes.4 A specific provision of the law covering testing provides that it shall not be “an unlawful employment practice for an employer to give and to act upon the results of any professionally developed ability test provided that such test, its administration or action upon the results is not designed, intended or used to discriminate because of race, color, religion, sex or national origin.”5

Case law suggests it is critical in defending exams to introduce evidence of formal validation studies that show that particular criteria predict performance.

Wards Cove Packing Co. v. Atonio6 illustrates how to determine if the challenged action had a disparate impact. In Wards Cove, Filipino and Alaska Native workers alleged they were affected by hiring practices of a salmon cannery. To determine if the impact was disparate, the court held it was proper to compare the racial composition of workers to the qualified population in the relevant market.

Due Process

A review of three cases illustrates issues that can arise when plaintiffs claim they were denied due process in taking an employment or credentialing test.

First, in Washington v. Davis,7 the plaintiffs alleged they were denied due process and equal protection under the Fifth and Fourteenth Amendments, respectively, because a verbal skills test for District of Columbia police recruits had a racially disproportionate impact on African Americans. The court of appeals had used the Title VII disparate impact analysis to evaluate the due process claim. But the Supreme Court said plaintiffs must show intentional discrimination and not just disproportionate racial impact to prevail on a Constitution-based due process claim. “[O]ur cases have not embraced the proposition that a law or other official act, without regard to whether it reflects a racially discriminatory purpose, is unconstitutional solely because it has a racially disproportionate impact.”

Second, in Mahmood v. Nat’l Bd. of Med. Exam’rs,8 a medical student was suspended from taking a medical licensing exam by the National Board of Medical Examiners (NBME) after she started a small fire in a restroom in the examination building in Maryland, where she was taking an exam that NBME administers. She was arrested and pled guilty to malicious destruction of property in connection with the fire. She alleged in her lawsuit, among other things, that NBME violated her Fourteenth Amendment right to procedural and substantive due process when it suspended her from taking the United States Medical Licensing Examination (USMLE) for three years. But due process claims may be brought only against government entities. The court held that NBME is not a state actor because there was no “close nexus” between NBME and Maryland, as the state played no role in developing NBME’s exams or applying disciplinary procedures to examinees who violate NBME policies. Furthermore, there was no “symbiotic relationship” between Maryland and NBME. The court held NBME acted independently in suspending Mahmood; it stated that what Maryland does with this information is a matter within the state’s discretion. Finally, the medical testing services provided by NBME were not an “exclusive state function.”

Third, in Golden Rule Life Ins. Co. v. Mathias9 the plaintiff successfully challenged an insurance agent licensing test. The court held that the Educational Testing Service (ETS) was a state actor because it was intertwined with the Director of Insurance of the State of Illinois, because it decided the format of test, and it graded the test and issued certificates. The court found the test contained inappropriate subject areas and had an arbitrary passing score and standards (unrelated to competency, confusing in form and structure). It held that ETS had provided inadequate guidance to applicants, did not conduct pretesting, and that a 50% pass rate was evidence that the test was simply a screening mechanism. The court held that the insurance company could challenge the agent licensing test under state and federal due process law.

Uniform Guidelines and Test Standards

A second broad area in which validity arises is uniform guidelines and test standards, which help establish test validity.

In Ricci v. Destefano,10 the city of New Haven relied upon a test to determine rankings for firefighter promotions. The city hired a private testing company to conduct a job analysis—that is, to identify essential tasks, knowledge, skills, and abilities for lieutenants and captains. A testing company developed a 100-question test and provided test preparation resources. Whites outperformed non-Whites on the test; the city threw out and refused to rely upon the test results due to concerns regarding the impact the results had on non-Whites. White and Hispanic firefighters sued over probable lost promotions. The Court held that intentional race-based action like New Haven’s—taken to avoid a disparate impact on a protected class—is impermissible under Title VII unless the employer can demonstrate that it had a “strong basis in evidence” that it would be subject to “disparate impact liability” if it failed to take the action.

Gulino v. Bd. of Educ. of N.Y.C.11 is a long-running class action involving teacher certification tests in New York. One test was held to have a disparate impact under Title VII on African American and Latino teachers. The courts found numerous problems with the exam, including an inadequate job analysis and no list of professional tasks or a determination of the knowledge needed to perform tasks. The test drafters chose the subtopics before collecting material and conducting interviews with subject matter experts, and therefore had insufficient evidence about the materials used to draft exam content.

In addition, the exam included content not directly related to teaching, and it was not created by professional test preparers. The school board failed to establish minimum knowledge requirements. The pretest population was not representative of the full population. Finally, the passing score was not related to job performance. The case is ongoing, with a court-appointed evaluator scheduled to recommend class certification decisions at the end of August 2022.

Passing Standards and Scoring Challenges

Validity issues also arise in the contexts of passing standards and scoring challenges.

Formal Adoption of Standards

Marquez v. Med. Bd. of California12 illustrates the importance of adopting formal standards. The California board used the Federation of State Medical Boards’ recommended passing score of 184 for one of the Step exams that make up the USMLE. A state statute required the board to set a passing score by resolution. However, the board never formally adopted such a resolution. The USMLE program administered the exam, but the board retained authority to set the passing score. One month before the plaintiff, an applicant taking the Step exam, took the exam for the fourth and final time, the board raised the passing score above 184. The applicant earned a 184 and challenged the failure to formally adopt a passing score. The court ordered the board to formally set a passing score and allow the applicant to retake the exam one more time.

Score Appeals

Turner v. Nat’l Council of State Bds. of Nursing13 underscores the need to have an appeals process. The Council told an applicant with dyslexia that if he passed with accommodations his license would be listed as “restricted and limited.” The applicant took the exam without accommodations and failed, alleging technical problems, which he claimed prevented him from completing the computerized test. He sued the State of Kansas under the Americans with Disabilities Act (ADA), claiming, among other things, that the Council failed to provide an appeals procedure. Although the appeals court said the plaintiff had standing to challenge the lack of an appeals process, it ruled he failed to link his problems with the exam to his dyslexia, and therefore, the trial court was correct to dismiss his claim.

Accommodations

Under the ADA,14 as enacted in 1990 and amended in 2008, a disability is defined as a physical or mental impairment that substantially limits a major life activity, having a record of such an impairment, or being regarded as having such an impairment.15 Individuals who are disabled within the meaning of the ADA may need accommodations to ensure that they can test in an accessible place and manner. This section considers representative cases concerning extra testing time and other accommodations, technology-related auxiliary aids, and score annotations for accommodated tests.

Early cases under the ADA tended to focus on whether a person was disabled, as that term had been interpreted in a fairly narrow manner by the US Supreme Court. Under the 2008 amendments to the ADA, Congress mandated a broader scope of coverage and made clear that the focus should be on whether a defendant discriminated—not on whether the person has a disability.

The court, in Jenkins v. Nat’l Bd. of Med. Exam’rs,16 illustrates the new focus required by the 2008 amendments. A third-year medical student sought extra time on the USMLE due to an alleged reading disorder. The court held his disability did not limit his ability to perform tasks central to most people’s daily lives because he was able to read, although slowly. The Sixth Circuit sent the case back to the court to consider the ADA Amendments Act, stating that “[t]he categorical threshold scope of the ADA’s coverage has been broadened.” The case later settled. As shown below, disputes over extra time often become a battle of experts.

Extra Time

Two cases are illustrative of the issues raised when an examinee claims they need extra time on an exam. First, in Kelly v. West Virginia Bd. of Law Exam’rs, a bar examinee with a severe learning disorder was granted time and half to complete the exam but failed twice.17 After filing suit under the ADA, the examinee underwent numerous cognitive tests that resulted in a diagnosis of a learning disability. His expert recommended double time, testifying that “[w]hat I’m asking is that Mr. Kelly be given as much time as he needs to show what he knows all the way to the end of the exam and see if he gets enough right to pass the exam.” A defense expert recommended time and a half (based on Mr. Kelly’s low average test results). The Fourth Circuit affirmed the board’s time-and-a-half accommodation.

In a second extra-time case, Currier v. Nat’l Bd. of Med. Exam’rs, a mother of a four-month-old baby requested 60 minutes of extra time per day to express breast milk during the USMLE.18 The board offered 45 minutes. The plaintiff’s expert said that this was insufficient. The Massachusetts Supreme Court ruled that lactating mothers are covered under the state equal rights act barring discrimination based on sex and ordered Massachusetts to offer 60 minutes of extra time. The plaintiff passed on her second attempt.

Annotating Scores

The next three cases concern score annotations, which is the practice of adding a “flag” or annotation to a score achieved with certain testing accommodations, or a note to explain or comment on the conditions under which an examinee achieved a score, including passing with accommodations. Over time, most testing programs have discontinued the practice of annotating test scores, in part because of the related litigation; therefore these cases are now largely of historic interest. The cases presented the question of whether accommodations altered the meaning of the resulting scores such that they were no longer comparable to scores achieved under standard testing conditions—thereby warranting an annotation so that score users would know that the scores of the accommodated test taker might have the same meaning as the score of someone who tested without extra testing time or other accommodations. 

In Doe v. Nat’l Bd. of Med. Exam’rs,19 a Pennsylvania medical student with multiple sclerosis challenged the annotation of his accommodated scores. The NBME provided the plaintiff with a score and, although it annotated the score, NBME did not state that the plaintiff’s scores were invalid. Psychometric experts testified that comparing accommodated scores with scores under standard conditions is not possible. The Third Circuit ruled that there is no requirement under the language of the ADA or its regulations that the score be declared psychometrically comparable to the scores of examinees who take the test under standard conditions. The court noted that “flagging,” or annotating, was not an inequitable imposition—because it does not state that an examinee’s scores are invalid. The court also held that the plaintiff had the burden to prove his scores were comparable to scores under standard conditions, and that he failed to carry that burden. Later, the court held the plaintiff could not show likelihood of future injury, as California (his intended jurisdiction of practice) had no discretion to treat scores differently and because residency programs are unlikely to use scores to compare candidates’ performance.

In the second annotations case, ­Breimhorst v. Educ. Testing Service, a plaintiff who did not have hands took the Graduate Management Admission Test (GMAT) on a computer with 25% additional time.20 ETS, which administered the GMAT at the time, added the following annotation: “Scores obtained under special conditions.” The plaintiff argued that the Federal Rehabilitation Act (1973) requires that the focus stay on the exam, rather than noting the accommodations. Similarly, the ADA requires that the “examination results accurately reflect” an individual’s ability, rather than disability.21 The district court held that ETS had the burden of proving that accommodated scores were not comparable to (i.e., did not have the same meaning as) scores achieved under standard testing conditions to lawfully annotate the accommodated scores. The court distinguished Education Department regulations, which permitted annotated scores if colleges do not deny admission due to nonstandard conditions. The court said the burden of proof was on the test provider to prove it “best ensured” that the test equally measured abilities of disabled and nondisabled test takers. ETS settled and discontinued the practice of annotating accommodated scores.

In the final annotation case, a federal court held that flagging scores might violate the ADA. Dept. of Fair Employment and Housing v. Law School Admission Council22 is a ­pattern-and-practice case (in which the Attorney General attempts to show that the defendant has systematically engaged in discriminatory activities) brought by a California state agency and joined by the US Department of Justice. As in Breimhorst, the court said that the testing entity had the burden to prove that accommodated test scores were not comparable and therefore warranted an annotation. It stated that flagging could be an ADA violation if it resulted in the Law School Admission Test (LSAT) not being accessible to disabled examinees, because flagging could be discriminatory, as it could intimidate examinees from requesting accommodations. The case settled, and the Law School Admission Council (LSAC) no longer annotates test scores.

Technology

As technology advances, so might the list of auxiliary aids that are reasonable as accommodations. This section considers three cases in which advances in screen reader technology led to a change in the law of accommodations.

First, in Elder v. Nat’l Conf. of Bar Exam’rs,23 a legally blind plaintiff requested JAWS (Jobs Access with Speech) screen-reader software to take the Multistate Bar Examination in California, rather than a live reader. The State Bar of California agreed, but NCBE opposed the requested accommodation. The court ordered NCBE to provide JAWS, ruling that it would “best ensure” that the test equally measured abilities of disabled and nondisabled test takers, quoting the Breimhorst case. (Interestingly, a federal district court in Maryland reached a contrary holding in an earlier case that Mr. Elder brought against NCBE, involving accommodations on the Maryland bar examination.) 

Second, in Jones v. Nat’l Conf. of Bar Exam’rs,24 the plaintiff had a visual impairment and learning disorders. She requested to test using ZoomText and Kurzweil 1000 screen-reader software. NCBE offered triple time and a menu of accommodations including Braille, audio CD, enlarged print, closed captioned TV, and a human reader—but not a screen reader. The court said that the menu of accommodations NCBE offered was insufficient. Instead, an individual analysis was required. The court applied the ADA’s “accessibility” and “best ensure” standards and required NCBE to provide ZoomText and Kurzweil 1000.

Finally, in Binno v. American Bar Ass’n, a legally blind applicant sued the American Bar Association (ABA) because it required law schools to use the LSAT as an admission factor, and one component of the exam—Analytical Reasoning—purportedly required diagramming to arrive at the answers, which blind examinees could not do.25 Because the ABA did not exercise control over the test administration, the court held that it could not be sued under the ADA for allegedly discriminating against the plaintiff. Holding that he should have sued LSAC instead, the court rejected the plaintiff’s ADA claims, and the US Supreme Court declined to hear the case. A separate lawsuit brought by Mr. Binno against LSAC was settled.

Security

Test security and validity intersect when considering issues of group invalidation and copyright infringement.

Group Invalidation

In Doe v. Nat’l Bd. of Podiatric Med. Exam’rs, five podiatry students acquired access to an exam and emailed exam content to a broader group of students during the four-day testing period.26 The board, having discovered this breach of security, invalidated all examinees’ scores from the period in question. The lead plaintiff, one of the students taking the exam during that testing period, passed—only two compromised questions were on his exam. There was no evidence that the plaintiff had access to the illicitly obtained questions. The court held it would defer to the National Board of Podiatric Medical Examiners as the test provider, if it considered relevant material submitted by the test taker and acted in good faith. Because the board held a hearing to consider the plaintiffs’ arguments, the court held it could invalidate all the scores from the administration at issue due to the security breach.

Copyright Infringement

The final security case concerns test preparation programs. In NBME v. Optima Univ.,27 NBME discovered low scores and unusual answers from examinees in Bucharest, Romania, and Budapest, Hungary, suggesting individuals were taking the exam to access the items, not to become doctors. Test center videos revealed examinees holding a device near the screen to photograph the questions. Further investigation, including sending an informant to Optima University’s study classes, revealed at least 50 copied questions. The court awarded statutory damages (for registered copyrights) of $2.4 million.

Conclusion

Test validity arises in a number of different case law contexts. Whether a test measures what it claims to measure often depends on whether the test is properly designed and administered, including under secure conditions. The cases in this article illustrate the need for testing organizations to apply best practices, legal and psychometric. Although legal challenges can succeed or fail, they have led to a better understanding of test validity in practice and can serve as a guide for creating and administering standardized tests.

_{* Based on an outline by attorney S.E. Phillips. More in-depth treatment of many of the cases are in Phillips’s Chapter 11, “Legal Issues for Credentialing Examination Programs” in Testing in the Professions: Credentialing Policies and Practice (Susan Davis-Becker and Chad W. Buckendahl eds., Routledge 2017).}

Contact us to request a pdf file of the original article as it appeared in the print edition.