Comparing Two Model Designs for Clinical Note Generation; Is an LLM a Useful Evaluator of Consistency?

Nathan Brake, Thomas SchaafNAACL 2024 · DOI

TL;DR

This paper explores two methods for automatically generating clinical SOAP notes from doctor-patient conversations: a single model generating the entire note versus separate models for each section. They found that generating sections sequentially (conditioned on previous ones) improves consistency, and importantly, demonstrated that LLMs like Llama2 can effectively serve as a proxy for human evaluators in assessing clinical note consistency, enabling scalable quality control.

What Problem Does This Paper Address?

Physicians are required to document patient encounters in Electronic Health Records (EHRs), often using the SOAP (Subjective, Objective, Assessment, Plan) note format. Automating this process can alleviate physician workload. However, generating accurate and consistent clinical notes, especially from audio recordings (DoPaCos), is challenging. Inconsistencies, such as conflicting patient ages or body part injuries across different sections of a note, can lead to errors in patient care and billing. Existing automatic metrics like ROUGE and Factuality don't fully capture these nuanced consistency issues. This paper addresses the need for robust methods to generate consistent clinical notes and, crucially, investigates whether Large Language Models (LLMs) can be leveraged to evaluate this consistency at scale, mimicking human judgment.

Methods

The authors compare two approaches for generating clinical notes using the PEGASUS-X Transformer model:

  1. General Model (GENMOD): A single PEGASUS-X model is fine-tuned to generate the entire SOAP note (specifically CC, HPI, and A&P sections) in one go. Special tokens are used to delineate sections, allowing for individual section scoring if needed.
  2. Specialized Model (SPECMOD): Three separate PEGASUS-X models are fine-tuned, with each model responsible for generating a single section (CC, HPI, or A&P). The final note is constructed by concatenating the outputs of these individual models.

The core hypothesis is that the GENMOD approach, by generating sections sequentially and implicitly conditioning on previous content, will lead to more internally consistent notes compared to the independent generation of sections in SPECMOD.

For evaluation, they employ:

  • Automatic Metrics: ROUGE (for summarization quality) and Factuality (for factual accuracy).
  • Human Evaluation: Five human reviewers (including medical experts) assess generated notes on four criteria: Age consistency, Gender consistency, Body Part consistency, and Coherence.
  • LLM Evaluation: The Llama2 LLM is used to evaluate the same criteria as human reviewers, with various prompting strategies (zero-shot, 1-shot, etc.) explored to align LLM performance with human judgment.

Datasets

The study utilizes a proprietary dataset of asynchronous doctor-patient conversations (DoPaCos). This dataset consists of approximately 10,000 training encounters, 500 validation encounters, and 500 test encounters, with an average conversation length of around 3,300 tokens. The paper notes that existing datasets like MediQA and ACI-Bench were not suitable for their specific experimental setup due to smaller sizes or different data characteristics (e.g., ambient clinical intelligence settings).

Key Insights and Ideas

  • Sequential Generation Boosts Consistency: Generating clinical note sections sequentially (GENMOD) significantly improves consistency in age, gender, and body part mentions compared to generating each section independently (SPECMOD). This highlights the importance of context and conditioning in generating coherent clinical narratives.
  • LLMs as Scalable Evaluators: LLMs, specifically Llama2, can serve as a valuable proxy for human evaluators in assessing specific clinical note consistency criteria (age, gender, body part). This is a crucial finding, as it allows for large-scale evaluation that would be prohibitively expensive and time-consuming with human reviewers alone.
  • Nuances in Consistency Evaluation: The study reveals that defining and evaluating "coherence" can be ambiguous, with different interpretations leading to varying results. This underscores the challenge of creating objective evaluation metrics for complex qualitative aspects of text.
  • LLM Prompting Matters: The effectiveness of LLM evaluation is highly dependent on the prompting strategy. While zero-shot and 2-shot prompting showed promising results, 1-shot prompting, due to its specific example, could inadvertently bias the LLM's evaluation.

Key Results

  • Automatic Metrics: GENMOD and SPECMOD showed very similar performance on ROUGE scores (less than 1% difference) and Factuality. This indicates that standard automatic metrics are insufficient to differentiate the consistency benefits of the GENMOD approach.
  • Human Evaluation: GENMOD outperformed SPECMOD in human evaluations for age (4.5% higher consistency), gender (2.5% higher), and body part (8.5% higher) consistency.
  • LLM Evaluation: Llama2 demonstrated strong agreement with human reviewers, achieving Cohen's Kappa scores of 0.79 for age, 1.00 for gender, and 0.32 for body part consistency when compared to a consensus human choice. This validates its utility as an evaluator.
  • Coherence Challenges: Both human and LLM evaluations showed moderate to minimal agreement on coherence, likely due to interpretation ambiguities.

Limitations and Future Work

  • ASR Quality Dependency: The models are trained on ASR transcripts, meaning the quality of the generated notes is inherently limited by the accuracy of the speech-to-text conversion. ASR errors can directly impact note quality.
  • Limited Human Reviewers: The human evaluation involved only 5 reviewers for a subset of notes, meaning individual ratings could have a disproportionate impact.
  • Specialty Bias: The LLM prompts were developed using a dataset with specific specialties. The LLM's performance might vary when evaluating notes from different or unseen specialties.
  • Future Work: The authors suggest expanding LLM-based evaluation to a wider range of criteria beyond consistency, developing more robust evaluation suites, and further exploring how generation and decoding strategies (like NRNS) impact note quality and consistency. They also leave the generation of the Objective section for future work.

Summary updated

BibTeX
@inproceedings{brake2024comparing,
  author = {Nathan Brake and Thomas Schaaf},
  title = {{Comparing Two Model Designs for Clinical Note Generation; Is an LLM a Useful Evaluator of Consistency?}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.findings-naacl.25},
}
Figure