Testing Task Force Quarterly Update

This article originally appeared in The Bar Examiner print edition, Winter 2019-2020 (Vol. 88, No. 4), pp 52–54.

For the latest Testing Task Force news—and to sign up to receive updates with the latest Task Force news, research, and blog posts—visit testingtaskforce.org.Testing Task Force Phase 2 Report Preview

Testing Task Force Phase 2 Report

The Testing Task Force is finalizing its comprehensive Phase 2 report on the nationwide practice analysis survey it developed and administered in 2019. The practice analysis was a huge endeavor involving careful research and planning that paid off by providing rich empirical data about the knowledge, skills, and abilities newly licensed lawyers (NLLs) need to perform their job activities. The Phase 2 report will be available on the Task Force’s website soon; a brief preview follows.

Developing the Practice Analysis Survey

NCBE completed a job analysis in 2012, but due to changes in the profession, the Task Force determined early in its study that it needed current job data to support its work and ensure that the bar exam continues to test relevant competencies for entry-level practice.

The development of the 2019 practice analysis survey happened between October 2018 and July 2019 in three consecutive stages:

1. An environmental scan of profession-­related information was done to draft lists of tasks typically performed by NLLs and the knowledge, skills, abilities, and other characteristics (KSAOs) necessary to perform those tasks, as well as a list of technologies used by NLLs to accomplish their work.
2. Focus groups of NLLs and supervisors of NLLs reviewed and edited the draft lists, which were then refined by the Task Force.
3. The practice analysis survey was pilot-tested to finalize the lists and survey structure.

Survey Design

The practice analysis survey was structured so that the survey data would identify core job tasks, KSAOs, and technologies that are important for all NLLs regardless of their practice specialties. (NLLs were defined as lawyers who have been licensed for three years or less.) The survey asked NLLs and attorneys who have, or have had, direct experience working with NLLs to rate lists of job tasks, KSAOs, and technologies required in practice. The job tasks were grouped into the four broad categories of General, Litigation/Dispute Resolution, Transactional/Corporate/Contracts, and Regulatory/Compliance, but were otherwise practice-area neutral.

The instructions and ratings scales were customized depending on whether the respondent was an NLL or not. To keep the survey to a reasonable length, respondents were randomly assigned to rate one of the four groups of items below:

• 49 General tasks and 24 technologies
• 74 Litigation/Dispute Resolution tasks
• 41 Transactional/Corporate/Contracts tasks and 36 statements of skills, abilities, and other characteristics (SAOs)
• 15 Regulatory/Compliance tasks and 77 areas of knowledge

Respondents rated the criticality and frequency of each task, the importance of each knowledge area, the criticality of each SAO, and the importance of proficiency in each technology. All respondents were presented with the same demographic questions, which included their primary practice setting, the number of lawyers in their organization, their gender and race, the jurisdiction(s) in which they are licensed, the jurisdiction where they have their primary practice, and their area(s) of practice.

Survey Administration

The practice analysis survey was administered from August 1, 2019, to October 2, 2019, by ACS Ventures LLC (ACS)—an independent psychometric consulting firm—using a web-based survey software. Over the course of the two months that the survey was open, the Task Force received assistance from bar associations and state Supreme Courts across 54 jurisdictions to inform attorneys around the country about the opportunity to participate. The survey was also publicized on the Task Force’s website and through its social media accounts.

Preview of Survey Results

A total of 30,970 people accessed the survey, and the total effective sample size of people who provided ratings was 14,846. (See table below). Respondents generally were representative of the overall population of lawyers based on a demographic comparison to data in the American Bar Association (ABA) Profile of the Legal Profession 2019.

Survey respondents by number of years licensed

Many more details and in-depth analysis of the results will be included in the forthcoming report. The following are a few of the responses.

• The most common areas of practice were Contracts; Business Law; Commercial Law; Administrative Law; Real Estate; Criminal Law; Torts; Employment Law and Labor Relations; Appellate; and Wills, Estates, and Trusts.
• Among the knowledge areas rated as important for NLLs were Rules of Professional Responsibility, Civil Procedure, Contracts, Evidence, and Legal Research Methodology, while skills and abilities that were rated as important included Reading Comprehension, Analytical Thinking, Written Expression, Identifying Issues, and Fact Gathering.
• The following tasks rated highest among respondents: Identify issues in client matter, including legal, factual, or evidentiary issues; Research case law; Interpret laws, rulings, and regulations for client; and Research statutory and constitutional authority.
• Word Processing Software, Electronic Communication Software, and Research Software or Platforms were the highest-rated technologies NLLs should be proficient in.

Phase 3

The practice analysis results, along with the feedback provided by stakeholders during the listening sessions in Phase 1 of the study, will be the basis for the final phase of the Task Force’s work, which will focus on program redesign. Phase 3 will consist of developing a test blueprint (detailing what content will be tested on the bar exam) and a test design (specifying how the content will be tested). The Phase 3 activities will be led by ACS, and stakeholder comments will be sought before the blueprint and design are finalized. More information about Phase 3 will be shared in the next update. In the interim, check the Task Force website for the Phase 2 report, which will be available soon (subscribers will be notified when it is published).

Survey respondents by jurisdiction

NOTE: Survey demographic questions were not mandatory; some respondents chose not to provide the jurisdiction in which they practice.

Contact us to request a pdf file of the original article as it appeared in the print edition.