The Testing Column: Joining NCBE as the New Director of Diversity, Fairness, and Inclusion Research

This article originally appeared in The Bar Examiner print edition, Summer/Fall 2021 (Vol. 90, Nos. 2–3), pp. 40–41.By Danette Waller McKinley, PhD

I am pleased to have the opportunity to introduce myself to the bar admissions community in this issue’s Testing Column. I joined the National Conference of Bar Examiners in April 2021 as Director of Diversity, Fairness, and Inclusion Research at a time when the organization is positioning itself for the next generation of the bar exam. My role allows me to participate in work on the new exam and its implementation and ensure that fairness is a central consideration in the decisions that are made during implementation.

Over the past few months, I have already had the privilege of working on several NCBE initiatives, such as the collaboration with the Council on Legal Opportunity Inc. (CLEO), in which NCBE provides funding and other support for the NCBE/CLEO Bar Passage Program to help CLEO law students prepare to pass the bar examination—a collaboration that supports NCBE’s and CLEO’s shared goal of increasing diversity and inclusion in the legal profession. I have also represented NCBE in discussions about the collection of demographic information, and about the aspects of diversity that those in legal education and bar admissions study.

Since I joined NCBE, I have been selected for participation in the Blueprint for Racial Justice, an initiative of the Conference of Chief Justices (CCJ) and the Conference of State Court Administrators (COSCA) that seeks to improve racial justice, equity, and inclusion in the justice system, and I am a member of its Increasing Diversity of the Bench, Bar, and Workforce Working Group, one of four working groups of the CCJ/COSCA initiative.

The most exciting aspect of my role with NCBE is the opportunity to manage the research agenda providing further evidence of the validity of NCBE examinations. Fairness is a central component of this agenda. Consideration of the unintended consequences of test results is an area of validity research of particular interest to me. Examples of the types of topics that such considerations include are examination of test use (e.g., the use of bar examination scores to evaluate the quality of law schools), the relationship of test results with other elements of bar admissions (e.g., character and fitness investigations), and the impact of test results on stakeholder interpretation (e.g., bar exam passage and disciplinary actions). The Diversity, Fairness, and Inclusion page on the NCBE website, ncbex.org/about/diversity-­fairness-and-inclusion/, highlights the work that has already been done by NCBE research staff in these areas, and this page will continue to be updated to reflect the ongoing work that we are undertaking.

I’d like to share a bit about my background and the development of my interest in the area of research on diversity and inclusion. My career journey started when I graduated from a liberal arts college with a BA in psychology and wondered what to do next. I started a master’s program and intended to become a reading specialist. In the first semester of the program, I took a course on diagnosing learning differences in reading. In studying the various measures used to help inform how to best assist children to improve their reading skills, I realized I wanted to know more about how those tests were developed.

I left the program and transferred to another master’s program that focused on educational research methodology in three areas: research design, statistics, and assessment. I found my home. I completed the two-year program and joined the National Board of Medical Examiners (NBME), where I conducted research on the licensure and certification of health professionals. While I was there, I gained valuable operational experience working on certification examinations for medical specialties. I was able to apply what I had learned about assessment as well as the research skills I had acquired.

This foundation was particularly important, because it allowed me to determine a focus for my doctoral work: licensure and certification testing. I was fortunate to gain experience in several aspects of assessment—data management, test calibration and scaling, and item writing and review—which allowed me to gain an understanding of the effort it takes to manage these processes. I was at NBME when they transitioned from a discipline-based examination to a more integrated examination that better reflected what physicians were expected to do in practice. The lessons I learned about stakeholder engagement, communication, and examination development were invaluable.

I also learned to identify research projects within my operational work. This applied research allowed me to contribute to the accumulation of validity evidence for the examinations I worked on. What characteristics of items and examinees contribute to differences in outcomes based on the mode of testing? Does it matter where pretest tasks are positioned on a performance-based assessment? How can you set the cut score for a performance task with a holistic rating scale? The answers to these questions helped to inform policy and addressed operational issues.

As part of these investigations, fairness was always a consideration. Test outcomes were studied to examine facets that could affect performance. In my position at ECFMG|FAIMER, our research team studied several aspects of an assessment that involved live interaction with a trained, standardized patient. (FAIMER, the Foundation for Advancement of International Medical Education and Research, is the nonprofit foundation for ECFMG, the Educational Commission for Foreign Medical Graduates.) Monitoring drifts in ratings, consistency in the use of the scoring rubric, and interactions between the characteristics of the cases and those of the examinees were part of the operational work that was done and that was disseminated through presentations and publications.

These are exciting times, and I am happy to be joining an organization that has “a competent, ethical, and diverse legal profession” as its vision. I am eager to apply the knowledge and experience I have gained over my career to contribute to the next generation of the bar exam in ensuring that the bar exam remains fair to all applicants.

Contact us to request a pdf file of the original article as it appeared in the print edition.