Abstract
Objective Structured Clinical Examinations (OSCEs) are widely used to assess medical students' communication skills, but scoring interview-based assessments is time-consuming and potentially subject to human bias. This study explored the potential of large language models (LLMs) to automate OSCE evaluations using the Master Interview Rating Scale (MIRS). We compared the performance of four state-of-the-art LLMs (GPT-4o, Claude 3.5, Llama 3.1, and Gemini 1.5 Pro) in evaluating OSCE transcripts across all 28 items of the MIRS under the conditions of zero-shot, chain-of-thought (CoT), few-shot, and multi-step prompting. The models were benchmarked against a dataset of 10 OSCE cases with 174 expert consensus scores available. Model performance was measured using three accuracy metrics (exact, off-by-one, thresholded). Averaging across all MIRS items and OSCE cases, LLMs performed with low exact accuracy (0.27 to 0.44), and moderate to high off-by-one accuracy (0.67 to 0.87) and thresholded accuracy (0.75 to 0.88). A zero temperature parameter ensured high intra-rater reliability (α = 0.98 for GPT-4o). CoT, few-shot, and multi-step techniques proved valuable when tailored to specific assessment items. The performance was consistent across MIRS items, independent of encounter phases and communication domains. We demonstrated the feasibility of AI-assisted OSCE evaluation and provided benchmarking of multiple LLMs across multiple prompt techniques. Our work provides a baseline performance assessment for LLMs that lays a foundation for future research into automated assessment of clinical communication skills.
Introduction
Objective Structured Clinical Examinations (OSCEs) represent the gold standard for evaluating the interpersonal and clinical communication skills of medical students. In these assessments, students interact with standardized patients—trained actors—while being evaluated on domains such as empathy, rapport-building, and information gathering. However, the traditional human-led grading process is resource-intensive, costly, and susceptible to rater fatigue and subjective bias. This study investigates the feasibility of leveraging Large Language Models (LLMs) to automate these evaluations using the validated 28-item Master Interview Rating Scale (MIRS).
Methodology
The researchers developed a multi-stage pipeline to convert OSCE video recordings into structured evaluations. The process began with audio extraction and transcription using OpenAI’s Whisper, followed by manual diarization to distinguish between the student ("Physician") and the patient.
Four distinct prompting strategies were evaluated:
- Zero-shot: Providing the rubric and transcript directly.
- Chain-of-Thought (CoT): Requiring the model to list evidence before assigning a score.
- Few-shot: Including up to five labeled examples of interactions.
- Multi-step: A two-stage process where the model first extracts relevant excerpts and then performs the final scoring.
The benchmark dataset comprised 10 unique OSCE cases, ranging from 7 to 30 minutes, with 174 expert consensus scores provided by medical educators at the University of Connecticut. All models were tested at a temperature of 0 to ensure deterministic results.
Results and Discussion
The study revealed that while LLMs struggle with exact score matching on a 5-point scale (exact accuracy: 0.27 to 0.44), they excel at "thresholding." The models demonstrated high accuracy (0.75 to 0.88) in distinguishing between students who meet proficiency (scores 3-5) and those who require additional support (scores 1-2). This suggests that AI could serve as an effective "red-flag" system for identifying students in need of remediation.
Claude 3.5 Sonnet emerged as the top performer, achieving an exact accuracy of 0.52 in optimal configurations. Furthermore, the models exhibited exceptional intra-rater reliability; GPT-4o achieved a reliability score of 0.98 at temperature 0, highlighting a level of consistency often unattainable by human graders.
Interestingly, advanced prompting techniques like few-shot and multi-step did not universally improve performance, occasionally introducing "noise" that degraded results. Additionally, multimodal attempts to score non-verbal cues directly from video using Gemini 1.5 Pro were unsuccessful, likely due to limitations in video resolution and frame composition.
Conclusion and Future Work
This research establishes a baseline for automated OSCE evaluation. While current models face challenges with holistic judgment and non-verbal cues, their high thresholded accuracy and consistency offer significant promise for medical education. Future work will focus on improving automated diarization, expanding datasets, and moving toward providing formative feedback to help students refine their clinical communication skills.
Summary updated