Benchmarking Generative AI for Scoring Medical Student Interviews in Objective Structured Clinical Examinations (OSCEs)

Jadon Geathers, Yann Hicke, Colleen Chan, Niroop Rajashekar, Sarah Young, Justin L. Sewell, Susannah Cornes, René F. Kizilcec, Dennis ShungLecture Notes in Computer Science 2025 · DOI · OpenAlex

Abstract

Objective Structured Clinical Examinations (OSCEs) are widely used to assess medical students' communication skills, but scoring interview-based assessments is time-consuming and potentially subject to human bias. This study explored the potential of large language models (LLMs) to automate OSCE evaluations using the Master Interview Rating Scale (MIRS). We compared the performance of four state-of-the-art LLMs (GPT-4o, Claude 3.5, Llama 3.1, and Gemini 1.5 Pro) in evaluating OSCE transcripts across all 28 items of the MIRS under the conditions of zero-shot, chain-of-thought (CoT), few-shot, and multi-step prompting. The models were benchmarked against a dataset of 10 OSCE cases with 174 expert consensus scores available. Model performance was measured using three accuracy metrics (exact, off-by-one, thresholded). Averaging across all MIRS items and OSCE cases, LLMs performed with low exact accuracy (0.27 to 0.44), and moderate to high off-by-one accuracy (0.67 to 0.87) and thresholded accuracy (0.75 to 0.88). A zero temperature parameter ensured high intra-rater reliability (α = 0.98 for GPT-4o). CoT, few-shot, and multi-step techniques proved valuable when tailored to specific assessment items. The performance was consistent across MIRS items, independent of encounter phases and communication domains. We demonstrated the feasibility of AI-assisted OSCE evaluation and provided benchmarking of multiple LLMs across multiple prompt techniques. Our work provides a baseline performance assessment for LLMs that lays a foundation for future research into automated assessment of clinical communication skills.

Introduction

Objective Structured Clinical Examinations (OSCEs) represent the gold standard for evaluating the interpersonal and clinical communication skills of medical students. In these assessments, students interact with standardized patients—trained actors—while being evaluated on domains such as empathy, rapport-building, and information gathering. However, the traditional human-led grading process is resource-intensive, costly, and susceptible to rater fatigue and subjective bias. This study investigates the feasibility of leveraging Large Language Models (LLMs) to automate these evaluations using the validated 28-item Master Interview Rating Scale (MIRS).

Methodology

The researchers developed a multi-stage pipeline to convert OSCE video recordings into structured evaluations. The process began with audio extraction and transcription using OpenAI’s Whisper, followed by manual diarization to distinguish between the student ("Physician") and the patient.

Overview of the AI-assisted OSCE evaluation pipeline, from video transcription to LLM scoring.

Four distinct prompting strategies were evaluated:

  1. Zero-shot: Providing the rubric and transcript directly.
  2. Chain-of-Thought (CoT): Requiring the model to list evidence before assigning a score.
  3. Few-shot: Including up to five labeled examples of interactions.
  4. Multi-step: A two-stage process where the model first extracts relevant excerpts and then performs the final scoring.

The benchmark dataset comprised 10 unique OSCE cases, ranging from 7 to 30 minutes, with 174 expert consensus scores provided by medical educators at the University of Connecticut. All models were tested at a temperature of 0 to ensure deterministic results.

Results and Discussion

The study revealed that while LLMs struggle with exact score matching on a 5-point scale (exact accuracy: 0.27 to 0.44), they excel at "thresholding." The models demonstrated high accuracy (0.75 to 0.88) in distinguishing between students who meet proficiency (scores 3-5) and those who require additional support (scores 1-2). This suggests that AI could serve as an effective "red-flag" system for identifying students in need of remediation.

Example of a medical interview transcript, the evaluation prompt, and the resulting LLM score with justification.

Claude 3.5 Sonnet emerged as the top performer, achieving an exact accuracy of 0.52 in optimal configurations. Furthermore, the models exhibited exceptional intra-rater reliability; GPT-4o achieved a reliability score of 0.98 at temperature 0, highlighting a level of consistency often unattainable by human graders.

Interestingly, advanced prompting techniques like few-shot and multi-step did not universally improve performance, occasionally introducing "noise" that degraded results. Additionally, multimodal attempts to score non-verbal cues directly from video using Gemini 1.5 Pro were unsuccessful, likely due to limitations in video resolution and frame composition.

Conclusion and Future Work

This research establishes a baseline for automated OSCE evaluation. While current models face challenges with holistic judgment and non-verbal cues, their high thresholded accuracy and consistency offer significant promise for medical education. Future work will focus on improving automated diarization, expanding datasets, and moving toward providing formative feedback to help students refine their clinical communication skills.

Summary updated

BibTeX
@inproceedings{geathers2025benchmarking,
  author = {Jadon Geathers and Yann Hicke and Colleen Chan and Niroop Rajashekar and Sarah Young and Justin L. Sewell and Susannah Cornes and René F. Kizilcec and Dennis Shung},
  title = {{Benchmarking Generative AI for Scoring Medical Student Interviews in Objective Structured Clinical Examinations (OSCEs)}},
  booktitle = {Lecture Notes in Computer Science},
  year = {2025},
  doi = {10.1007/978-3-031-98420-4\_17},
}

Loading related content...

Figure