Automating Evaluation of AI Text Generation in Healthcare with a Large Language Model (LLM)-as-a-Judge

Emma Croxford, Yanjun Gao, Elliot First, Nicholas Pellegrino, Miranda Schnier, John Caskey, Madeline Oguss, Graham Wills, Guanhua Chen, Dmitriy Dligach, Matthew M. Churpek, Anoop Mayampurath, Frank Liao, Cherodeep Goswami, Karen K. Wong, Brian W. Patterson, Majid AfsharmedRxiv 2025 · DOI

TL;DR

This paper introduces an "LLM-as-a-Judge" framework to automate the evaluation of AI-generated clinical summaries from Electronic Health Records (EHRs). This approach achieves high reliability comparable to human experts, significantly reducing evaluation time and cost, making it a scalable solution for ensuring the safety and accuracy of AI in healthcare.

What Problem Does This Paper Address?

Clinicians are often overwhelmed by the sheer volume of information in Electronic Health Records (EHRs). Large Language Models (LLMs) offer a promising solution by automatically summarizing patient data into actionable insights. However, ensuring the accuracy, precision, and safety of these AI-generated summaries is critical, especially in healthcare. Human expert evaluation is the gold standard but is time-consuming and expensive. This paper addresses the need for an efficient and reliable automated method to evaluate these clinical summaries.

Methods

The core of this research is the "LLM-as-a-Judge" framework. This approach uses an LLM to act as an evaluator, assessing the quality of AI-generated clinical summaries. The LLM is provided with the original patient notes, the generated summary, and a detailed evaluation rubric (the PDSQI-9 instrument). The study explored various LLM prompting strategies, including zero-shot, few-shot, supervised fine-tuning (SFT), direct preference optimization (DPO), and multi-agent frameworks. They also investigated the impact of LLM architecture (reasoning vs. non-reasoning models) and training methods on evaluation performance.

Datasets

The study utilized data from the original PDSQI-9 study, which included:

  • Original patient notes: From inpatient and outpatient encounters at the University of Wisconsin Hospitals and Clinics.
  • LLM-generated summaries: Created from these patient notes.
  • Human expert evaluation scores: Provided by seven physician raters using the PDSQI-9 instrument.

For cross-task validation, they used data from the Problem List BioNLP Summarization (ProbSum) 2023 Shared Task, which involved summarizing patient diagnoses from the MIMIC-III EHR database.

Key Insights and Ideas

  • LLMs can be reliable judges: The study demonstrates that LLMs, particularly those with reasoning capabilities, can effectively evaluate clinical summaries with high inter-rater reliability, comparable to human experts.
  • Reasoning matters for evaluation: LLMs that incorporate reasoning capabilities (like GPT-03-mini) outperform non-reasoning models in evaluating complex attributes like "Cited," "Organized," and "Synthesized," which require deeper understanding and synthesis of information.
  • Speed and scalability: LLM-as-a-Judge is orders of magnitude faster than human evaluation (22 seconds vs. 600 seconds), offering a significant advantage for large-scale deployment.
  • Domain-specific challenges: Evaluating clinical summaries presents unique challenges (hallucinations, omissions) that require more nuanced approaches than general-purpose LLM evaluation benchmarks.

Key Results

  • GPT-03-mini (5-shot) achieved the highest ICC of 0.818, demonstrating strong agreement with human evaluators.
  • Evaluations were completed in an average of 22 seconds by GPT-03-mini, a 96% reduction compared to human evaluators.
  • Reasoning models showed superior performance across most PDSQI-9 attributes, especially those requiring advanced comprehension.
  • Cross-task validation on the ProbSum task confirmed high reliability, indicating the generalizability of the LLM-as-a-Judge approach.
  • Multi-agent frameworks showed potential for mimicking human variability by assigning distinct personas, though they did not outperform single-agent models in terms of ICC.

Limitations and Future Work

  • External validation: The study primarily used data from UW Health and MIMIC-III. Further validation in other health systems is needed.
  • Task specificity: The validation focused on summarization tasks. Extension to other clinical language generation tasks (e.g., medical question answering) requires additional work.
  • Computational costs: Training and running some LLMs, especially for multi-agent frameworks, can be computationally expensive.
  • Infrastructure availability: The use of HIPAA-secure cloud environments might not be universally available to all health systems.
  • Future work: Exploring other human-aware loss functions for training smaller models and extending the LLM-as-a-Judge approach to other clinical NLP tasks are suggested.

Summary updated

Abstract

Electronic Health Records (EHRs) store vast amounts of clinical information that are difficult for healthcare providers to summarize and synthesize relevant details to their practice. To reduce cognitive load on providers, generative AI with Large Language Models have emerged to automatically summarize patient records into clear, actionable insights and offload the cognitive burden for providers. However, LLM summaries need to be precise and free from errors, making evaluations on the quality of the summaries necessary. While human experts are the gold standard for evaluations, their involvement is time-consuming and costly. Therefore, we introduce and validate an automated method for evaluating real-world EHR multi-document summaries using an LLM as the evaluator, referred to as LLM-as-a-Judge. Benchmarking against the validated Provider Documentation Summarization Quality Instrument (PDSQI)-9 for human evaluation, our LLM-as-a-Judge framework demonstrated strong inter-rater reliability with human evaluators. GPT-o3-mini achieved the highest intraclass correlation coefficient of 0.818 (95% CI 0.772, 0.854), with a median score difference of 0 from human evaluators, and completes evaluations in just 22 seconds. Overall, the reasoning models excelled in inter-rater reliability, particularly in evaluations that require advanced reasoning and domain expertise, outperforming non-reasoning models, those trained on the task, and multi-agent workflows. Cross-task validation on the Problem Summarization task similarly confirmed high reliability. By automating high-quality evaluations, medical LLM-as-a-Judge offers a scalable, efficient solution to rapidly identify accurate and safe AI-generated summaries in healthcare settings.
BibTeX
@misc{croxford2025automating,
  author = {Emma Croxford and Yanjun Gao and Elliot First and Nicholas Pellegrino and Miranda Schnier and John Caskey and Madeline Oguss and Graham Wills and Guanhua Chen and Dmitriy Dligach and Matthew M. Churpek and Anoop Mayampurath and Frank Liao and Cherodeep Goswami and Karen K. Wong and Brian W. Patterson and Majid Afshar},
  title = {{Automating Evaluation of AI Text Generation in Healthcare with a Large Language Model (LLM)-as-a-Judge}},
  year = {2025},
  doi = {10.1101/2025.04.22.25326219},
  howpublished = {medRxiv},
}
Figure