Testing Task Force Quarterly Update

For the latest Testing Task Force news—and to sign up to receive updates with the latest Task Force news, research, and blog posts—visit www.testingtaskforce.orgTask Force Research Plan Announced

The Testing Task Force is pleased to announce that the research plan for its comprehensive, future-­focused study of the bar exam is now available. The study is supported by the Task Force’s two independent research consulting firms—ACS Ventures LLC and American Institutes for Research (AIR). ACS is a psychometric consulting firm that focuses on test design, operational support, and quality assurance within the credentialing, education, and workforce sectors. AIR is a nonprofit behavioral and social science research organization that specializes in analyzing job requirements.

The study will proceed in three overlapping phases, with each phase building on the previous ones. The first and second phases of the study are already under way. The study will consider the content, format, delivery method, and timing of the bar examination and the MPRE, and it will be done collaboratively, with input from stakeholders solicited throughout the study.

The goal of the Task Force’s study is to develop a set of recommendations for the NCBE Board of Trustees that is supported by the research data gathered by its consultants and that takes into consideration logistical and psychometric requirements along with stakeholder input to lay the groundwork for the next generation of the bar examination.

Phase 1: Stakeholder Research

The initial phase of the study, conducted by ACS, will focus on gathering stakeholder input about the bar examination. During this phase, ACS will conduct a series of focus groups and listening sessions with stakeholders (e.g., bar admission agencies, state supreme courts, bar leaders, legal educators, law students, practitioners) to gather information about uses and perceptions of the bar examination and ideas and opinions about the future direction of the bar examination in a changing legal profession. This phase began in November 2018 and will overlap with the beginning of Phase 2.

Phase 2: Future-Focused Practice Analysis

Some results of the first phase of the study will help inform the second phase, a future-focused practice analysis conducted by AIR to identify the job activities (tasks) of newly licensed lawyers (NLLs) and the knowledge, skills, abilities, and other characteristics (KSAOs) required to perform them. The practice analysis will be comprehensive and systematic, based on multiple sources of job knowledge (e.g., new practitioners, supervisors, legal subject-matter experts) and using multiple methods to obtain input (e.g., focus groups and surveys). It will address both the current state of the profession and expected changes to the profession in the coming years to accurately identify the critical knowledge, skills, and abilities required for competent entry-level practice now and in the future.

The practice analysis will consist of five steps, each designed to allow multiple opportunities for subject-­matter experts to provide input and to build on the work done in the preceding steps. The first step is already under way.

1. Conduct environmental scan. AIR will review the results of the 2012 NCBE practice analysis (a study completed by Applied Measurement Professionals Inc. on behalf of NCBE to describe the activities of newly licensed lawyers) and other available studies, reports, articles, books, websites, and online databases to develop an initial list of tasks of NLLs and the KSAOs needed to perform those tasks.
2. Conduct focus groups. AIR will conduct multiple focus groups to ask practitioners and experts about changes to the field and the future direction of the profession. AIR will also work closely with various subject matter experts, NLLs, and supervisors to refine the list of tasks and KSAOs to develop the future-focused practice analysis survey.
3. Administer future-focused practice analysis survey. AIR will develop an updated inventory of tasks and KSAOs informed by the first two steps and will administer a nationwide web-based survey presenting this inventory to licensed lawyers in their first five years of practice, and their supervisors, for their input. Survey respondents will provide ratings on job tasks and KSAOs that will help determine criticality to entry-level practice.
4. Conduct linkage exercises. AIR will conduct meetings with subject-matter experts to link the critical tasks identified in the survey results to the corresponding KSAOs needed to complete each task. The purpose of the linkage exercise is to identify which KSAOs are necessary for completing the most critical work entry-level lawyers perform.
5. Prepare practice analysis technical report. Finally, AIR will prepare a technical report that documents its methodology, analyses, and final list of critical tasks and KSAOs. The report and the results from this study will provide the foundation for decisions regarding any future changes to the bar examination and its test specifications.

We anticipate that this phase of the study will be completed in early 2020.

Phase 3: Bar Examination Program Design and Test Components Design

In this phase of the project, ACS will use the information collected in the first two phases to develop multiple options for program design, taking into consideration stakeholder perspectives and needs, logistical issues, best practices in high-stakes testing, and fidelity to the practice of law (representation of the professional framework of expected competencies). ACS will lead an iterative process where the Testing Task Force reviews and considers program design options and solicits comments from stakeholders. The program design plan is expected to delineate the number of exam components, the domains that each component will cover, and the expected path/relationship among the components (e.g., pass exam A before exam B, domain-specific versus general skills). ACS will then use the program design to develop recommendations for test components design. Test components design involves decisions about the measurement format, test administration plans, and scoring strategy. The test components design will present a structured plan for the development, delivery, and maintenance of each proposed test component. We anticipate that this final phase of the study will be completed by the fall of 2020.

As the study progresses, the Task Force will publish summaries and reports of findings on its website. The Task Force expects that its study will produce useful information for the benefit of all, including data and insights about core competencies that could be adopted by other stakeholders such as law schools, bar associations, employers, and others involved in ensuring that newly licensed lawyers are prepared to practice effectively and safely.