TL;DR
This paper introduces a novel three-step pipeline for generating reliable, evidence-grounded medical question answers from electronic health records (EHRs). By combining LLM-based evidence classification, guided response generation, and iterative quality control, the system achieves strong factuality and faithfulness, addressing critical challenges in automating clinical communication.
What Problem Does This Paper Address?
The healthcare industry faces a significant challenge in efficiently managing patient messages, which often requires clinicians to sift through extensive electronic health records (EHRs) to craft personalized and accurate responses. This time-consuming task contributes to clinician burnout and can potentially impact patient care quality. While Large Language Models (LLMs) show promise in automating this process, they are prone to factual errors (hallucinations) and may struggle to accurately interpret complex medical contexts. Furthermore, traditional evaluation metrics like BLEU and ROUGE often fail to correlate well with human judgment, making it difficult to ensure the reliability and safety of AI-generated medical responses. This paper tackles the need for a robust, transparent, and verifiable system for generating clinically grounded medical answers from EHRs.
Methods
The authors propose a structured three-step pipeline:
- Evidence Classification: LLMs are used to classify EHR evidence segments into three categories: relevant, supplementary, or not relevant. This is framed as a multiple-choice task, with categorical labels constrained by Enum types for consistency. The LLM is prompted to provide a rationale before its final label selection to improve interpretability and faithfulness.
- Grounded Question Answering & Quality Control: The LLM generates clinician-facing responses based on the classified evidence. These responses are then subjected to a systematic quality assurance process that checks for structural consistency, citation accuracy, and length. Deficiencies trigger iterative feedback prompts to the LLM for revision and improvement, enforcing adherence to clinical communication standards.
- Comprehensive Evaluation: The system's performance is evaluated using a two-phase strategy. The first phase benchmarks foundational medical reasoning on standard multiple-choice datasets. The second phase involves a comprehensive evaluation of the entire pipeline on the ArchEHR-QA dataset, assessing evidence classification, response generation, and automated quality control. A novel "LLM-as-a-Judge" methodology, specifically a claim-level triple extraction method, is introduced to evaluate faithfulness by breaking down claims into subject-predicate-object triples and verifying their support in the EHR evidence.
Datasets
- Evaluation Datasets:
- ArchEHR-QA: The primary dataset used for evaluating the end-to-end pipeline, focusing on grounded question answering from EHRs.
- MMLU-Pro-Med, MedQA-US, MedMCQA, PubMedQA: Used for benchmarking general medical knowledge assessment.
Key Insights and Ideas
- Structured Pipelines for Reliability: The paper emphasizes that a structured, multi-stage pipeline (classification, generation, quality control) is crucial for achieving reliable and transparent LLM-based medical QA, rather than relying on end-to-end generation alone.
- "LLM-as-a-Judge" for Faithfulness: The introduction of a claim-level triple extraction method for faithfulness evaluation is a significant contribution. This approach offers a more scalable and interpretable way to assess whether generated statements are truly supported by the source evidence, moving beyond simple lexical overlap.
- Iterative Refinement is Key: The inclusion of an iterative feedback loop for response revision highlights the importance of continuous quality improvement in LLM-generated content, especially in high-stakes domains like healthcare.
- Trade-offs Between Extractive and Generative Models: The authors acknowledge that traditional metrics might favor extractive methods, while generative models offer fluency. Their approach aims to bridge this gap by ensuring generative outputs are grounded in evidence.
Key Results
- General Medical Knowledge: GPT-4o achieved state-of-the-art performance on MMLU-Pro-Med with 81.56% accuracy and strong scores on other medical knowledge benchmarks.
- Grounded Question Answering (ArchEHR-QA):
- GPT-4o achieved 82.0% claim-level evidence faithfulness on the test set.
- GPT-4o achieved 51.6% citation-level factuality (strict citation F1) on the test set.
- GPT-4o also demonstrated strong performance in relevance and faithfulness on development sets.
Limitations and Future Work
- Data Leakage in Benchmarks: A significant limitation identified is the potential for data leakage in existing medical knowledge benchmarks, where commercial LLMs might have inadvertently been trained on benchmark content, inflating performance metrics. This highlights the need for greater dataset transparency or explicit controls for training-evaluation separation.
- Lack of Expert Validation: The study did not include formal expert validation of the model-generated responses due to time constraints. While the LLM-as-a-Judge framework provides a scalable assessment, human expert oversight is still considered essential for confirming clinical accuracy and safety.
- Future Work:
- Integrating external knowledge retrieval with interactive clinical tools.
- Examining the impact of these systems on clinician workload.
- Conducting comprehensive human preference studies to align evaluation frameworks with clinician expectations.
- Further research into distinguishing between claim faithfulness and citation faithfulness.
- Improving label consistency and evaluation framework design.
Summary updated