TL;DR
This paper introduces WisPerMed, a three-part system for answering patient questions using Electronic Health Records (EHRs). By focusing on relevance and grounding answers in evidence, WisPerMed significantly improved performance on the ArchEHR-QA 2025 challenge, outperforming the baseline and demonstrating a path towards more trustworthy AI in healthcare.
What Problem Does This Paper Address?
As more patients gain access to their EHRs, the volume of questions directed at healthcare providers is skyrocketing. While Large Language Models (LLMs) can generate fluent responses, they are prone to "hallucinations" – fabricating information – which poses a serious risk to patient safety. The ArchEHR-QA 2025 shared task specifically addresses this by requiring AI systems to answer patient questions using EHR data, but critically, these answers must be grounded in the provided evidence and include explicit citations. This means not only understanding the question and finding relevant information but also ensuring the generated answer is factually accurate and directly traceable to the source documents.
Methods
WisPerMed tackles this challenge with a modular, three-part pipeline:
- Relevance Classification (BioClinicalBERT): The first step involves identifying which sentences in a patient's clinical notes are relevant to a given question. They fine-tune a BioClinicalBERT model for this task, classifying sentences as "essential," "supplementary," or "not-relevant." To improve robustness, they augment the training data with synonyms and paraphrased versions of sentences.
- Grounded Answer Generation (DistilBART-Med-Summary): Once relevant sentences are identified, a second model, DistilBART-Med-Summary, generates an answer. Crucially, this model is constrained to use only the top-ranked relevant evidence identified in the first step, ensuring the answer is directly supported by the source material.
- Citation Alignment (BiomedBERT): To ensure transparency and trustworthiness, the final stage aligns each sentence in the generated answer back to its specific supporting evidence in the clinical notes. This is achieved by embedding answer sentences and evidence sentences using BiomedBERT and then calculating similarity scores (ROUGE) to establish explicit citations.
Datasets
The primary dataset used is the ArchEHR-QA dataset. This dataset is derived from the MIMIC-III database and consists of patient questions, clinician-rewritten questions, and annotated clinical note excerpts. Sentences within the clinical notes are manually labeled as "essential," "supplementary," or "not-relevant" for answering specific questions. The paper also mentions using UMLS and WordNet for synonym augmentation and Gemini for paraphrase generation.
Key Insights and Ideas
- Modularity is Key: Breaking down the complex task into distinct stages (relevance, generation, citation) allows for focused optimization and better control over the output.
- Relevance-First is Crucial: Prioritizing the identification of relevant evidence before generating an answer is fundamental to producing accurate and grounded responses.
- Data Augmentation Pays Off: Expanding the training data with synonyms significantly boosted the performance of the relevance classifier, highlighting the importance of robust sentence understanding.
- BART Models Excel at Grounded Generation: The experiments suggest that BART-based models are better suited for this specific task of grounded EHR question answering compared to models like Flan-T5.
- Transparency Builds Trust: Explicitly linking answer sentences to their source evidence is vital for building trust in AI-generated medical information.
Key Results
WisPerMed achieved an overall score of 35.0% on the ArchEHR-QA 2025 hidden test set, which was 4.3 percentage points higher than the organizer's baseline. Key improvements were noted in:
- BERTScore: A 44% relative improvement, indicating better semantic alignment with reference texts.
- SARI: A 119% improvement, suggesting enhanced content preservation during text simplification or rewriting.
- Overall Factuality and Relevance: Both metrics saw substantial improvements, demonstrating the effectiveness of the relevance-first and grounded generation approach.
Limitations and Future Work
The authors acknowledge several limitations:
- Citation Recall Cap: The reliance on hard probability thresholds in the relevance classifier limits citation recall to around 27%.
- Data Augmentation Nuances: Gemini paraphrasing occasionally introduced label noise, and synthetic data had a minor impact on overall metrics.
- Generalizability: Models tuned on MIMIC data might perform less well on different institutional note formats or specialized jargon.
- ROUGE Similarity: The ROUGE-score-based similarity method for citation alignment can sometimes misassign identifiers for semantically similar sentences.
They suggest that more advanced Retrieval-Augmented Generation (RAG) approaches, which jointly optimize retrieval and generation, could potentially outperform their current three-part pipeline and further reduce citation errors. Future work could also explore more sophisticated methods for citation alignment and investigate the performance of their system on a wider range of EHR data.
Summary updated