Consultation Checklists: Standardising the Human Evaluation of Medical Note Generation

Aleksandar Savkov, Francesco Moramarco, Alex Papadopoulos Korfiatis, Mark Perera, Anya Belz, Ehud ReiterEMNLP 2022

TL;DR

Evaluating automatically generated medical notes is notoriously difficult due to subjective expert opinions. This paper introduces "Consultation Checklists" (CCs) as a structured, expert-crafted ground truth to standardize this evaluation process. CCs significantly improve inter-annotator agreement and the correlation of automatic metrics with human judgments, offering a more objective and scalable approach to assessing medical note generation systems.

What Problem Does This Paper Address?

The generation of medical notes from patient consultations is a critical task in healthcare, aiming to reduce clinician workload and burnout. However, evaluating the quality of these automatically generated notes is a significant challenge. Existing methods, including human evaluation and automatic metrics like ROUGE and BLEU, suffer from subjectivity, high cost, and poor reproducibility. A key issue is the inherent subjectivity in medical note generation: experts often disagree on which facts are important, whether they should be included, and how they should be weighted. This leads to low inter-annotator agreement, making it hard to reliably assess and compare different note generation systems.

Methods

The core contribution of this paper is the Consultation Checklist (CC) protocol. This protocol involves two main stages:

  1. Creation Stage: An expert clinician listens to a consultation audio recording and creates a structured, itemized list of all facts discussed. This list, the Consultation Checklist, includes both relevant and irrelevant information and is marked for clinical importance (Critical, Non-critical, Irrelevant). This detailed, fact-based approach aims to provide a more objective "ground truth" than a single human-written note.
  2. Evaluation Stage: Human raters (clinicians) use the pre-created Consultation Checklist to evaluate system-generated notes. They mark each statement in the generated note as correct/incorrect and critical/non-critical/irrelevant, and then mark which items from the checklist are present or absent in the generated note. Crucially, evaluators do not need to listen to the consultation audio during this stage, relying solely on the CC for reference. This process allows for the calculation of Precision and Recall based on the CC.

The paper also explores using CCs as references for automatic metrics like ROUGE and BERTScore, comparing their correlation with human judgments against using human-written notes as references.

Datasets

  • Mock Consultation Dataset: The paper utilizes a mock consultation dataset from Papadopoulos Korfiatis et al. (2022) for creating Consultation Checklists and for some of the evaluation studies.
  • Real-life Patient Consultations: For the main evaluation study, 20 real-life patient consultations were used.
  • BART-based Model: A BART encoder-decoder transformer model was used to generate History and Examination notes for evaluation. This model was pre-trained on CNN/Dailymail and fine-tuned on a proprietary dataset of 10,000 consultation transcripts and human-written notes.

Key Insights and Ideas

  • Subjectivity is the Enemy: The paper strongly emphasizes that the inherent subjectivity in defining what constitutes a "good" medical note is the primary obstacle to reliable evaluation.
  • Checklists as Objective Anchors: Consultation Checklists, by itemizing facts and their importance, act as a more objective "ground truth" than a single human-written note, which can be influenced by individual writing styles or priorities.
  • Granularity Matters: Breaking down facts into atomic items within the checklist allows for more precise evaluation and identification of specific errors.
  • Reusability of Checklists: Once created, CCs can be reused to evaluate multiple generated notes, making the evaluation process more scalable and cost-effective over time.
  • Automatic Metrics Benefit from CCs: Using CCs as references for automatic metrics like BERTScore significantly improves their correlation with human judgments compared to using human-written notes.

Key Results

  • High Inter-Annotator Agreement: The CC protocol achieved good levels of inter-annotator agreement in a pilot study for creating CCs (e.g., 93.7% fact coverage between two annotators). In the evaluation stage, the agreement scores (Krippendorff's Alpha) for Present/Absent and Correct/Incorrect were significantly higher than those reported in previous work using other methods.
  • Improved Automatic Metric Correlation: Using Consultation Checklists as references for automatic metrics (ROUGE, BERTScore, Levenshtein Distance) led to increased correlation with human judgments. BERTScore showed the highest gain, with a 20.5% increase in Spearman's correlation.
  • Time Efficiency: While creating a CC takes about an hour for a 10-minute consultation, the evaluation stage using CCs is relatively efficient, taking clinicians about 45 minutes per evaluation task (including listening to audio for verification). This is comparable to or better than previous methods.
  • Accuracy Trade-off: A small percentage of generated note items (3.91%) required changes to their correctness judgment after evaluators listened to the consultation audio, highlighting a trade-off between the standardization of CCs and capturing all nuances.

Limitations and Future Work

  • Creation Effort: Creating Consultation Checklists is time-consuming and requires expert clinicians, which is a bottleneck for scalability. The paper notes that while the first CC might take 30 minutes, subsequent ones take longer, and clinicians need breaks.
  • Nuance Capture: There's a trade-off between the conciseness of a CC and its ability to capture every subtle nuance of a consultation. This was demonstrated by cases where a generated note item was marked incorrect based on the CC but correct after listening to the audio.
  • Domain Specificity: The protocol was tested only on primary care consultations. The authors postulate that it could be applied to other medical domains (secondary care, etc.) but this remains to be tested.
  • Future Work: The authors suggest that the CC protocol could be generalized and applied to other domains beyond medical notes. Further research could focus on optimizing the CC creation process or exploring semi-automatic methods for checklist generation.

Summary updated

BibTeX
@inproceedings{savkov2022consultation,
  author = {Aleksandar Savkov and Francesco Moramarco and Alex Papadopoulos Korfiatis and Mark Perera and Anya Belz and Ehud Reiter},
  title = {{Consultation Checklists: Standardising the Human Evaluation of Medical Note Generation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
}
Figure