Evaluation and Development of Pathways to Legal Licensure

Issues, Considerations, and Questions for Jurisdictions Considering Changes to Admission Through Assessment and Admission on Motion

This article originally appeared in The Bar Examiner print edition, Summer 2024 (Vol. 93, No. 2), pp. 31-35.

Issues, Considerations, and Questions in Developing a Legal Licensure Pathway

Validity

1. The purpose of any licensure-­related assessment is to provide meaningful information regarding the knowledge, skills, abilities, judgments, and/or other characteristics of individuals seeking admission into a profession. Stakeholders charged with regulating admission use this information as part of the admissions decision for a given person at a particular point in time. Both the decision itself and the information used to make the decision should be defensible; formal validation work provides the basis for defensibility through “articulating the claims and assumptions associated with the proposed decision (the interpretation/use argument), empirically testing these assumptions, and organizing evidence into a coherent validity argument.”1 The evidence-based nature of modern validity theory in educational measurement is reflected through the issues, considerations, and questions listed in this document.
2. Is the assessment linked in some way to a “job analysis” or a “practice analysis” and, if so, how closely linked is it to the knowledge, skills, abilities, or judgments most commonly needed in entry-level practice?2
3. Are the criteria used to make the pass/fail decision supported by evidence?3
4. Does the pathway for assessment (e.g., work product submitted for review) represent an individual’s independent efforts (as opposed to reflecting, to at least some degree and to potentially varying degrees, others’ contributions)? If not, what are the implications, if any, for the granting of a nonprovisional, general license to an individual?
5. What evidence is available regarding the relationships between assessment evaluations or decision outcomes and other relevant measures of legal knowledge, skills, and abilities? For example, a validity study might look at the relationship between law school GPA and scores earned on a portfolio examination used for a licensure decision. Assuming that law school grades and portfolio assessments, for example, both measure one or more aspects of a person’s legal knowledge and skills (albeit in somewhat different ways), we might assume a general positive correlation between those two measures. To the extent that additional measures of the same core knowledge and skills are available to groups overseeing the licensure process, such measures should be employed in research studies to evaluate the assessment being used to make the licensure decision.
6. Does the pathway assess a broad or narrow range of knowledge and skills? To the extent that the range is broad, the assessment can more reasonably be used for a general license. To the extent that the range is narrow, the assessment is likely more appropriate for a somewhat limited license (i.e., a license limited to those specific knowledge areas and skills an individual demonstrated via the assessment.) This question could be considered in connection with additional requirements, such as a set of educational requirements. To the extent that there is ongoing oversight of curricula that contribute to assessing depth and/or breadth of knowledge and skills, there may be less need to establish depth or breadth within the assessment.
7. Are there any potential sources of bias associated with the pathway, and if so, do they have a substantial impact on the decisions being made? “Bias” in this context could take many forms. For example, if evaluations within the pathways are not anonymized, is there the potential for personal characteristics such as age, gender, race, ethnicity, native language, socioeconomic status, or other personal and irrelevant factors to influence those evaluations? If those conducting the evaluations (e.g., supervisors) know the individuals they are evaluating personally (e.g., as family members), what safeguards are in place to help ensure personal relationships or other conflicts of interest do not influence the evaluations?

Decision Consistency

1. Decision consistency refers to an estimate, usually a statistical estimate, of how likely it is that a candidate for licensure would have the same outcome (i.e., receiving a license or failing to receive a license) if the candidate underwent a similar (but not necessarily identical) assessment with no change to their knowledge, skills, abilities, or judgments. This is key for fairness and, in traditional testing contexts such as an examination, this consistency might be measured as reliability. Where possible, reliability should be estimated and reported.4 In other assessment contexts (e.g., pathways that do not include a traditional test), efforts should be made to meaningfully evaluate the likely effect that factors such as having different supervisor(s), or different professor(s), or having attended one law school versus another, for example, would have had on a person’s pass/fail outcome, if any. It is important that evidence related to the assessment program’s decision consistency be documented and formally evaluated. With a given pathway, how will decision consistency or reliability be assessed? Will there be a minimum level of consistency or reliability tolerated for pathways?
2. When a jurisdiction offers multiple pathways, the consistency of decisions both within and across those pathways should, to the extent possible, be evaluated on an ongoing basis. Will it be possible to do such ongoing evaluation? Who will be responsible for monitoring this, particularly if there is a minimum level of consistency or reliability required for the pathways? What will happen if a pathway fails to meet a minimum level of consistency?

Accessibility

1. “Accessibility” as a term is often used in the context of giving maximally equal opportunities to individuals with disabilities. It may also be used more broadly to refer to the extent to which it is possible for anyone who would like to participate in a program or pathway to do so. On the one hand, adopting multiple pathways might be seen as maximizing the opportunity for any individual to find a pathway that is a good fit for them, thereby increasing fairness within a licensure system. On the other hand, if not all pathways are fully open to all takers due to additional inherent challenges for individuals with disabilities and/or due to limited supervisory opportunities, for example, reduced fairness or perceptions of fairness could be an unintended consequence.
2. Is it possible for everyone who wants to participate in a pathway to do so? If not, how will limited opportunities be fairly distributed? For example, will everyone wanting to participate in an examination pathway be able to participate, or will a limited number of “seats” or testing locations restrict access to the pathway? In the case of a supervised practice pathway, are there enough supervisors for every candidate, or are spaces limited? If candidates are responsible for securing their own supervisor, how is fairness of opportunity ensured? If supervision opportunities are limited, what steps and resources are required to ensure fairness and transparency in who receives supervision?
3. For candidates with physical or mental impairments that substantially limit a major life activity (and who are therefore protected under the Americans with Disabilities Act), what accommodations will be offered/available? Who will decide whether an individual receives an accommodation? Who will provide any necessary technology, tools, or additional accommodations? If extra time is required to complete an assessment or complete individual pieces of work contributing to an assessment, who will determine how much additional time is needed and reasonable?5

Responsibility, Compliance, and Oversight

1. Who will conduct the assessment and make the recommendation/decision about whether to admit individuals based on their results?
2. Will another group or entity oversee or audit the procedures and results?
3. Will the assessment process involve specific curricular requirements? If so, who determines those requirements?
4. Which entity or entities (professors, law schools, boards of bar examiners, etc.) will be responsible for reporting the data needed for evaluations of decision consistency and to whom will they report it?
5. Will the assessment program comply with broadly accepted sources of best practices in licensure, including the Standards for Educational and Psychological Testing;6 the Code of Fair Testing Practices in Education;7 the Rights and Responsibilities of Examinees;8 and the Department of Education Office for Civil Rights?9 Which entity or entities will evaluate compliance?
6. To the extent that actual clients (as opposed to actors, for example) are used as part of the assessment, informed consent for participation should be gathered and identities and identifying information should be removed from work products. How will client privacy be protected in pathways that involve real-world work products?

Cost and Resource Requirements

1. What are the costs and resource requirements associated with developing and implementing the program? Who bears the costs and provides the resources?
2. What are the costs and resource requirements associated with administering the program? Who bears the costs and provides the resources?
3. What are the costs and resource requirements associated with ongoing monitoring and evaluation of the program? Who bears the costs and provides the resources? What costs are associated with revision if monitoring finds flaws?

Other Fairness-Related Considerations

1. According to the most recent edition of the Standards for Educational and Psychological Testing, “The term fairness has no single technical meaning and is used in many different ways in public discourse.”10 That said, impartiality is a bedrock principle of fairness in both educational measurement and the law.
2. Conflicts of interest and ­perceived conflicts of interest should be avoided.
   • Care should be taken in setting rules regarding supervisor qualifications and potential conflicts of interest (e.g., when supervisors, assessors, or evaluators are related by blood or marriage or otherwise have a close relationship with the individual being assessed).
   • Some credentialing organizations have advocated for separation between certification-granting agencies and education or training functions. Some law school faculty have recognized that they should serve a fully separate role as educators from the role bar examiners serve. This perspective tends to be driven by the belief that formative assessment is inherently different than summative assessment.11 It also recognizes the potential for conflicts of interest associated with having individuals working closely with students being the ones to make, or directly contribute to making, the final licensure decision.
3. To the extent that random assignment and/or anonymized grading are practical possibilities, they should be built into the assessment. To the extent that they are not practical possibilities, efforts should be made to collect independent evaluations from multiple highly trained and monitored assessors.
4. Each type of assessment has advantages and disadvantages. The trade-offs should be understood and, ideally, balanced. This balance could be achieved within the pathway itself or via other licensure requirements within a jurisdiction, such as graduation from an accredited law school, experiential learning requirements in or beyond law school, and/or continuing education requirements or evaluations.
   • Traditional examinations are indirect measures of likely performance in legal practice and therefore can feel inauthentic to the examinee. However, if carefully designed and administered, they offer objective and impartial grading, high reliability, and a broad sampling of knowledge and skills in a relatively short amount of time.
   • Direct observation of practice is more authentic, and fidelity to actual practice generally makes the assessments feel more meaningful and worthwhile. However, this model’s greater complexity makes consistent scoring more challenging. Reliability is generally lower,12 and the sampling of knowledge and skills is often narrower; if the range is not narrower, the assessment requires considerably more time.
   • Simulation exercises can represent a good compromise and help balance the trade-offs between the two other approaches. However, if simulations are not developed and scored with care, they can also represent the worst of both.
5. Policies and procedures relating to review of assessments or decisions should be developed. Is there a process available to a candidate seeking to challenge their score or decision outcome?13

Considerations for Jurisdictions in Admitting Candidates Licensed via Varying Pathways in Other Jurisdictions

Review of Current Rules

1. Is admission on motion currently permitted?
2. Is admission on motion granted for regular admission or limited admission?
3. Some jurisdictions’ rules specify that candidates must have been “admitted by bar examination.” Jurisdictions may include the word “examination” in the name they assign to a pathway but not include a traditional test within that pathway. Will licensure pathways that include the word “examination” but have few traditional components of a high-stakes examination be accepted under the rules?
4. Some jurisdictions’ rules limit the number of times a candidate for admission on motion may have previously failed a bar examination. Will this rule be applied to all pathway attempts? If a pathway includes “interim” assessments with multiple (or unlimited) opportunities to revise and resubmit work for reevaluation, what shall count as an attempt and/or a “failure”?
5. Some jurisdictions require in-person hearings for at least some attorneys seeking admission on motion. Will the method by which the attorney seeking admission on motion was originally licensed affect whether such a hearing is required or the purpose of those hearings? Will jurisdictions require other evidence of a candidate’s knowledge, skills, abilities, judgments, etc., in support of granting a general license to practice law in a given jurisdiction?

Development of New Rules and/or Clarification of Existing Rules

1. For jurisdictions that have rules regarding the length of time an applicant must have previously practiced, will supervised practice count toward the time limit?
2. Some jurisdictions have developed or are developing pathways that have an experiential education component, and this may be completed while a candidate is still in law school. Will this practice experience count toward a minimum number of years of practice requirement?

Contact us to request a pdf file of the original article as it appeared in the print edition.