UNIBUC-SD at ArchEHR-QA 2025: Prompting Our Way to Clinical QA with Multi-Model Ensembling

Dragos Ghinea, Ștefania RîncuACL 2025

TL;DR

This paper presents a resource-efficient method for clinical question answering (QA) from electronic health records (EHRs) using readily available, smaller language models. By employing multi-prompt ensembling and careful prompt engineering, the authors achieve strong performance without the need for expensive fine-tuning or specialized hardware, demonstrating the potential of accessible LLMs for clinical applications.

What Problem Does This Paper Address?

Clinicians face a significant workload responding to patient inquiries, often requiring them to sift through extensive electronic health records (EHRs) to find relevant information. Automating this process, specifically by generating answers grounded in clinical evidence from EHRs, is a crucial challenge. The ArchEHR-QA 2025 shared task aims to address this by evaluating systems on their ability to accurately detect relevant evidence and then summarize it into a coherent answer. The core difficulty lies in efficiently and accurately extracting and synthesizing information from complex, often lengthy, medical texts to answer specific patient questions.

Methods

The authors tackle this problem by breaking it down into two main subtasks: paragraph relevance classification and summarization.

  1. Paragraph Relevance Classification:
    • They use a multi-model ensembling approach, combining predictions from smaller, pre-trained models like Gemma and Mistral.
  2. Instead of asking models to classify the entire list of paragraphs at once, they reformulate the task as a binary classification for each paragraph. This significantly improved consistency and interpretability.
  3. Crucially, they rely on prompt engineering rather than fine-tuning. They experimented with various prompts, including Chain-of-Thought (CoT) prompting, to guide the models. CoT involves instructing the model to "think step-by-step" before providing an answer, which generally improved performance, especially for larger models.
  4. Majority voting is used to aggregate the relevance judgments from the ensemble of models.
  5. Summarization:
    • For summarization, they use a unified approach where the model generates a concise answer (around 70 words) directly from the identified relevant paragraphs, including inline citations.
  6. Gemma models were found to be particularly reliable for this task due to their consistent adherence to formatting requirements.
  7. To enforce the word limit, they implemented a multi-turn prompting strategy, re-prompting the model to shorten its output if it exceeded the limit.

A key aspect of their methodology is the use of quantized models to fit them onto consumer-grade hardware (a single RTX 5090) and the avoidance of computationally intensive fine-tuning.

Datasets

The primary dataset used for training and evaluation is from the ArchEHR-QA 2025 shared task. This dataset provides clinical notes, patient-authored questions, and clinician-formulated questions, along with indexed paragraphs from the notes. The authors specifically used the clinician-formulated question for relevance assessment to reduce false positives.

Key Insights and Ideas

  • Accessible LLMs Can Be Powerful: The paper strongly advocates for the effectiveness of smaller, readily available LLMs (like Gemma and Mistral) when combined with sophisticated prompt engineering and ensembling. This challenges the notion that only massive, resource-intensive models can achieve high performance in complex domains like clinical QA.
  • Prompt Engineering is Key: The authors demonstrate that carefully crafted prompts, especially those incorporating Chain-of-Thought reasoning, can significantly boost model performance without requiring any model updates. This highlights prompt engineering as a critical skill for leveraging LLMs.
  • Binary Relevance Classification is More Robust: Reformulating the relevance task as a binary decision for each paragraph, rather than a multi-label classification of the entire set, led to more consistent and interpretable results.
  • Ensembling for Robustness: Combining multiple models, even smaller ones, through majority voting improves the overall robustness of the system, helping to mitigate individual model weaknesses and reduce false positives/negatives.
  • Summarization Benefits from Relevance Filtering: The study confirmed that filtering for relevant paragraphs before summarization significantly improves the quality of the final answer, underscoring the importance of the relevance detection step.

Key Results

  • The proposed approach achieved a +20.2 Factuality and +4.9 Relevance gain over the baseline on the test set.
  • On the dev dataset, their best-performing ensemble achieved an overall factuality score of 60.4 and relevance of 35.8, significantly outperforming the baseline (43.1 factuality, 28.7 relevance).
  • The system was developed and tested using consumer-grade hardware (1x RTX 5090), demonstrating its practicality and cost-effectiveness.
  • While Chain-of-Thought prompting improved performance for larger models, smaller models sometimes struggled to complete the task reliably, generating only reasoning text.
  • The summarization component, while generally adhering to requirements, sometimes struggled with strict length constraints, requiring multi-turn prompting.

Limitations and Future Work

  • Model Size: The study focused on smaller, accessible models (up to 27B parameters) due to hardware constraints. Larger, state-of-the-art models might offer further improvements but at a significantly higher computational cost.
  • Quantization Impact: The use of quantized models, necessary for fitting them onto the hardware, might introduce a minor performance degradation compared to full-precision versions.
  • Prompt Engineering Sensitivity: The system's performance is heavily reliant on prompt engineering. Adapting the system to different QA formats or clinical contexts might require extensive re-tuning of prompts.
  • Dataset Specificity: The evaluation was conducted on a specific dataset. Generalization to other EHR datasets or real-world clinical deployments needs further validation.
  • Summarization Brittleness: The summarization component showed some brittleness in handling strict length constraints and citation completeness, requiring multi-turn prompting and postprocessing.
  • Evaluation Focus: The authors acknowledge that their internal optimization efforts disproportionately focused on the automatic factuality score, potentially leaving other important metrics (like BLEU, ROUGE) less optimized. A complete assessment of clinical utility would require evaluation by domain experts.

Future work could explore larger models, more advanced ensembling techniques, and broader validation across different clinical datasets.

Summary updated

BibTeX
@inproceedings{ghinea2025unibuc,
  author = {Dragos Ghinea and Ștefania Rîncu},
  title = {{UNIBUC-SD at ArchEHR-QA 2025: Prompting Our Way to Clinical QA with Multi-Model Ensembling}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
}
Figure