TL;DR
This paper presents UniBuc-SB, a system for generating grounded answers to patient questions from electronic health records (EHRs) under strict computational constraints. The system achieves competitive performance by leveraging pre-trained language models and a novel ensemble approach for relevance classification, demonstrating that effective EHR QA is possible even with limited resources.
What Problem Does This Paper Address?
The increasing use of patient portals has led to a surge in patient inquiries, overwhelming clinicians. Automating the process of answering these questions using Electronic Health Records (EHRs) is crucial. However, developing such systems faces significant challenges: limited datasets for training, the complexity of medical language requiring nuanced understanding, and the need for accurate information retrieval to ensure grounded, factual answers. This paper specifically tackles the ArchEHR-QA shared task, which requires generating answers directly from EHR excerpts, highlighting the need for systems that are both accurate and efficient, especially in resource-constrained environments.
Methods
The UniBuc-SB system employs a modular pipeline consisting of three main stages: preprocessing, relevance classification, and answer generation.
- Preprocessing: Raw EHR data and clinician questions are structured. Each medical case is divided into sentences, labeled as essential, supplementary, or irrelevant to the query. This stage ensures consistent data handling and prepares it for downstream tasks.
- Relevance Classification: This is a critical step to filter out irrelevant information. The system uses an ensemble of four pre-trained language models (BERT, Bio_ClinicalBERT, BlueBERT, and MedEmbed) fine-tuned for binary relevance classification. This ensemble approach leverages both general and domain-specific medical knowledge to improve robustness. To handle class imbalance (many irrelevant sentences), they use focal loss.
- Answer Generation: Relevant sentences identified in the previous stage are compiled and formatted with their identifiers. This evidence is combined with the clinician's question to create a prompt for the Mistral Small 3.1 language model. The model generates a concise answer, with a mechanism to iteratively refine it if it exceeds a word limit (75 words). A final grounding step adds citations to the generated answer, referencing the evidence sentences.
Datasets
- ArchEHR-QA Development Dataset: Consisted of 20 distinct medical cases.
- ArchEHR-QA Test Dataset: Consisted of 100 medical cases.
The authors explicitly state they refrained from incorporating any external datasets, relying solely on the limited training data supplied by the organizers.
Key Insights and Ideas
- Ensemble Power: Combining multiple pre-trained language models, even those with general domain knowledge like BERT, significantly boosts relevance classification performance compared to individual models. This suggests that general-domain representations can provide valuable complementary context.
- Resource Efficiency is Achievable: The system successfully operates within a 24GB VRAM limit using a single NVIDIA RTX 4090 GPU, demonstrating that high-quality EHR QA is feasible without extensive computational resources.
- Focal Loss for Imbalance: Using focal loss effectively addresses the class imbalance inherent in relevance classification tasks, where irrelevant sentences heavily outweigh relevant ones.
- Iterative Refinement for Conciseness: The two-step answer generation process (initial generation followed by length enforcement if needed) ensures that answers are both informative and adhere to strict length constraints.
Key Results
- The UniBuc-SB system achieved competitive scores in citation-based evaluation, with strict F1 scores of 44.7 (micro) and 46.4 (macro).
- The full ensemble of four models achieved the highest lenient F1 score of 0.619 in the ablation study.
- The system's inference pipeline runs comfortably within the 24GB VRAM limit, processing each case in approximately 16 seconds end-to-end.
- While factuality was strong, metrics like BLEU (0.6) and BERTScore (23.9) indicated room for improvement in the fluency and linguistic alignment of generated text.
Limitations and Future Work
- Computational Constraints: The reliance on a single RTX 4090 GPU limited the scope of model experimentation and hyperparameter optimization.
- Dataset Size: The small development dataset (20 cases) restricted the diversity of clinical scenarios encountered during training, potentially impacting generalization.
- Future Work: The authors suggest exploring methods for enhancing response generation, including advanced grounding techniques and multi-sentence contextual modeling, to improve fluency and linguistic alignment. They also aim to address false positives in sentence selection, which sometimes occur due to over-reliance on clinical terminology rather than true contextual relevance.
Summary updated