Behind the Scenes at NCBE’s Grading Workshop: A Photo Essay

This article originally appeared in The Bar Examiner print edition, September 2017 (Vol. 86, No. 3), pp 20–25.

In its commitment to assist jurisdictions in grading the written portion of the bar exam, NCBE prepares thorough grading materials for the Multistate Essay Examination and the Multistate Performance Test, which effectively guide graders through the grading process. In addition, NCBE holds an MEE/MPT Grading Workshop at its Madison, Wisconsin, headquarters the weekend following the bar exam—a one-day event that begins with a presentation by NCBE staff on grading fundamentals, followed by dedicated hands-on grading sessions for each MEE and MPT item. The grading sessions are led by drafting committee members who are familiar with the questions, grading materials, and content areas and who are experienced Grading Workshop facilitators.

During the Grading Workshop, participants read, grade, and discuss actual examinee answers from jurisdictions across the country. The Grading Workshop prepares graders for grading their own jurisdictions’ papers by familiarizing them with the questions and grading materials, identifying answer trends, refining grading weights, and discussing with other graders how to resolve grading issues. Participants also get a head start on calibration—the development of coherent and identifiable grading judgments so that rank-ordering is consistent throughout the grading process as well as across multiple graders. Discussion at the Workshop focuses additionally on assessing the quality of the writing in the examinee papers.

Participation in NCBE’s Grading Workshop has expanded over the years, with hundreds of graders representing most MEE and MPT jurisdictions participating in the Grading Workshop in person, by conference call, or via on-demand streaming following the workshop. The recent July 2017 Grading Workshop set a record for all three categories of participants, and we took the opportunity to prepare this photo essay to offer a look behind the scenes at staff preparation for the Workshop and at what occurs during the Workshop’s hands-on sessions.

The titles of NCBE staff featured in this photo essay are as follows:

• Judith Gundersen, NCBE President and CEO (holding the title of Director of Test Operations at the time of the Workshop)
• Joanne Kane, Ph.D., Associate Director of Testing
• Sonja Olson, MEE/MPT Associate Program Director
• Andrew Mroch, Ph.D., Research Psychometrician
• Ellen Embertson-Merrill, Educational Programming Coordinator
• Mike Petterson, Executive Assistant to the COO
• Ingrid Jorgenson, Test Production Coordinator
• Gina Bailey, Senior Test Associate
• Lu Riese, Senior Test Associate
• Kasey Brinkman, Test Associate
• Katie Chu, Test Associate

Preparation for the Grading Workshop

Registration for the Grading Workshop is handled by NCBE’s Educational Programming Coordinator, who sends confirmations and instructions to registrants specific to their type of participation—in person, by conference call, or via on-demand video—and arranges hotel logistics and prepares registration materials for in-person attendees.

Ellen Embertson-Merrill

• NCBE Test Operations Department staff members contact jurisdictions to request that they send in a selection of examinee answers after test administration to be used in the Workshop; those answers are then sorted, redacted for identifying information, and approved by NCBE’s Director of Test Operations and MEE/MPT Associate Program Director for use in Examinee Answer Books for Workshop participants.
• Examinee Answer Books—which include the item, scoring guide, examinee answers, and analysis for MEE items—are assembled, bound, and boxed for transportation to the in-person ­participants by NCBE Test Operations Department staff. Those participating by conference call or later via on-demand video are provided with instructions for secure online access to the Examinee Answer Books.

• Two pilot programs were launched at the July 2017 Workshop to enhance the interactive experience of graders. Poll Everywhere (a secure real-time audience response system that gathers live responses via texting, web, and a mobile app) was used in all sessions for graders to submit their grades—replacing the prior method of texting their grades to a staff member or relaying their grades verbally—allowing quicker tabulation of grades and providing anonymity for graders when submitting their grades. Zoom (a secure cloud-based video conferencing product allowing graders to join the session in a variety of ways) was used in two sessions and allowed graders participating by conference call to connect with the facilitator in a live, virtual environment much like attending the live session in person. Also new this year was an app for the Grading Workshop itself. These were all tested extensively by NCBE staff members prior to launch, and mock Zoom sessions were offered to graders so that they could familiarize themselves with the format and various ways to connect.

Mock Zoom Session

Mike Petterson testing the pilot programs and app

Friday, July 28: The Day Before the July 2017 Workshop

NCBE staff and facilitators gather at the conference hotel to finalize preparations for the Workshop.

Facilitators Jon Carlson and Shelly Kurtz (both Iowa), Mike Petterson

Workshop attendees, facilitators, and NCBE staff members attend a reception hosted by NCBE on Friday evening.

Frankie Jones (AZ)

Rebeka Fortess (NH), Sonja Olson

Ted Tollefson and Matt Gunn (both ID)

Martin Sjolie (WA)

Saturday, July 29: The Day of the July 2017 Workshop

NCBE staff members arrive at the conference hotel early on Saturday morning.

Ellen Embertson-Merrill, Ingrid Jorgenson, Katie Chu, Lu Riese, Kasey Brinkman, Mike Petterson, Gina Bailey

Judy Gundersen, Joanne Kane

Sonja Olson, facilitator Shelly Kurtz (IA), Gina Bailey, Mike Petterson, Lu Riese, Katie Chu

Staff preparation includes setup for videotaping the sessions for later on-demand viewing.

Saturday morning begins with a presentation on grading fundamentals by three NCBE staff members.

Judy Gundersen

Joanne Kane, Andy Mroch

Seven facilitators help lead the 10 hands-on sessions that take place throughout Saturday morning and afternoon. (Not pictured: Alexander Scherr [GA], who facilitated remotely.)

Facilitators Adisa Harrington (TX) and Catherine Klein (DC)

Facilitators Jon Carlson and Shelly Kurtz (both IA)

Facilitator Alan Palmiter (NC)

Facilitator Judy Gundersen

Sessions begin with an overview of the item and grading materials, and any questions about the area of law (MEE) or assigned task (MPT) are addressed.

Participants then silently read the real examinee answers collected from jurisdictions across the country.

Participants grade the answers.

Staff members in each session assist the facilitators with compiling the attendees’ grades.

Facilitator Alan Palmiter (NC), Lu Riese

Gina Bailey, facilitator Shelly Kurtz (IA)

Participants and facilitators discuss the grades assigned, with graders being called on to explain the grades they gave—particularly if a grade might be an outlier from grades assigned by other graders in the session.

Based on the review and grading of the sample of examinee answers and the ensuing discussion between graders and facilitators, grading materials may be refined or grading weights adjusted. The final versions of these grading materials are then made available to graders in user jurisdictions after the Workshop.

All sessions are videotaped for on-demand viewing.

The July 2017 MEE/MPT Grading Workshop had a record number of participants:

Attendees from Kansas: Patrick Hughes, Donald Peterson, Fran Angell, David Cooper, Peter Johnston (Kansas attendee Molly Wood not pictured)

Attendees from Hawaii: Front row: Annette Chock, Hon. Adrianne Heely; Back row: Shellie Park-Hoapili, Leighton Hara, Lester Oshiro

Attendees from New Mexico: Front row: Twila Hoon, Sophie Martin, Bonnie Stepleton, Henry Alaniz; Back row: Olga Serafimova, Hon. Edward Chavez, Michael Thomas, Kevin Hammar, Karen Grohman, Michael Hoses (New Mexico attendee Michelle Haubert not pictured)

Contact us to request a pdf file of the original article as it appeared in the print edition.