Human Evaluation and Correlation with Automatic Metrics in Consultation Note Generation

Francesco Moramarco, Alex Papadopoulos-Korfiatis, Mark Perera, Damir Juric, Jack Flann, Ehud Reiter, Anya Belz, Aleksandar SavkovACL 2022 · DOI

Introduction

Clinician burnout has reached a crisis point in modern healthcare, largely fueled by the administrative burden of Electronic Health Records (EHR). A primary contributor to this workload is the requirement to write detailed SOAP (Subjective, Objective, Assessment, Plan) notes following every patient interaction. While automated abstractive summarization—or Note Generation—offers a potential solution, the field lacks a clinical "gold standard" for evaluation.

Standard NLP metrics like ROUGE or BLEU, originally designed for news summaries, often fail to capture clinical safety. In a medical context, linguistic fluency is secondary to factual accuracy; a single hallucinated fact or the omission of a critical negative (e.g., "no chest pain") can have serious consequences. This study investigates the degree to which automatic metrics correlate with actual clinical utility and the "post-editing" effort required by medical professionals.

Methodology

The researchers developed a rigorous, multi-stage human evaluation framework to assess AI-generated clinical notes. The process began with fine-tuning 10 BART-based summarization models on a proprietary dataset of 130,000 real-world consultations. These models were tasked with generating the "Subjective" portion of SOAP notes.

The multi-stage workflow for dataset creation and human evaluation.

Five practicing General Practitioners (GPs) evaluated 57 mock consultations from the Primock57 dataset. The evaluation involved four distinct tasks: listening to consultations to write original notes, post-editing five note versions (four AI-generated and one human-written) for factual accuracy, quantifying errors (incorrect statements and omissions), and providing qualitative feedback on structural issues like repetition.

The Gap Between Metrics and Utility

A central finding of the study is the "Reference Bias" inherent in automatic metrics. These metrics correlate significantly better with post-edited versions of a note than with the original human-written reference. This occurs because post-edited notes maintain the same surface-level vocabulary and sentence structure as the AI output, whereas different humans may describe the same medical facts using entirely different linguistic forms.

Surprisingly, the study found that simple metrics often outperformed complex ones. Character-based Levenshtein distance—a straightforward measure of the number of edits required to transform one string into another—performed as well as, or better than, sophisticated contextual models like BERTScore. Because Levenshtein distance directly measures edit operations, it serves as a more accurate proxy for a clinician's actual workload and post-editing time.

Key Results

The quantitative analysis revealed a clear performance gap between human-written and AI-generated notes. While AI-generated notes are currently less efficient than human notes, they still provide a baseline that may be faster to edit than writing a note from scratch.

Comparison of human-written and AI-generated notes across key evaluation criteria.

As shown in the results, post-editing a human-written note took approximately 96.5 seconds, compared to 136.4 seconds for AI-generated notes. The AI notes also exhibited a higher frequency of errors, with "Repetition" (93 instances) and "Incorrect Statements" (34 instances) being the most common qualitative issues identified by the evaluators.

Conclusion and Future Work

This research highlights that Levenshtein distance, BERTScore, and METEOR are currently the most suitable metrics for evaluating automated consultation notes. However, a significant limitation remains: current metrics cannot distinguish between minor errors (e.g., a wrong date for a meal) and critical clinical errors (e.g., missing a family history of colon cancer).

Future efforts must focus on developing "medically-weighted" metrics that prioritize clinical safety over linguistic fluency. Additionally, moving beyond mock consultations to real-world clinical data will be essential to capturing the inherent complexity and "messiness" of actual patient-doctor interactions.

Summary updated

BibTeX
@inproceedings{moramarco2022human,
  author = {Francesco Moramarco and Alex Papadopoulos-Korfiatis and Mark Perera and Damir Juric and Jack Flann and Ehud Reiter and Anya Belz and Aleksandar Savkov},
  title = {{Human Evaluation and Correlation with Automatic Metrics in Consultation Note Generation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.acl-long.394},
}
Figure