UTSamuel at ArchEHR-QA 2025: A Clinical Question Answering System for Responding to Patient Portal Messages Using Generative AI

Samuel Reason, Liwei Wang, Hongfang Liu, Ming HuangACL 2025

TL;DR

This paper introduces UTSamuel, a clinical question answering system that leverages generative AI (GPT-4o) to automatically draft responses to patient portal messages. By grounding answers in electronic health records and citing evidence, UTSamuel aims to reduce clinician burden and improve patient communication efficiency.

What Problem Does This Paper Address?

Patient portals have become a crucial communication channel, but they also create a significant workload for clinicians. The sheer volume of messages, often dealing with complex medical issues, contributes to clinician burnout. Manually responding to these messages can consume hours daily, extending work beyond clinic hours. This paper addresses the need for an automated system that can assist clinicians by generating accurate, evidence-based responses to patient queries directly from their electronic health records (EHRs). The ArchEHR-QA 2025 shared task specifically focuses on this challenge, emphasizing grounded question answering with direct citation of supporting evidence.

Methods

The UTSamuel system employs a three-step pipeline:

  1. Sentence Selection: The system identifies "essential" sentences that directly answer the patient's question (e.g., diagnosis, treatment) and "supplementary" sentences that provide necessary context (e.g., medications, lab results). Sentences deemed "not relevant" are excluded. This selection process was refined through an iterative self-evaluation approach using ChatGPT (GPT-4o) on a development set, where the model was prompted to analyze its own sentence selection logic against human-annotated relevance labels. The final logic focuses on understanding the clinical question's intent and classifying sentences accordingly.
  2. Answer Generation: A structured prompt is used to instruct GPT-4o to generate a medically accurate, concise (under 75 words), and clear response. This prompt includes the patient's question, the selected essential and supplementary sentences, and explicit instructions to cite each sentence using its unique ID.
  3. Citation Formatting: Each generated sentence in the response is appended with its corresponding sentence ID from the EHR, ensuring traceability and grounding.

Crucially, the system was developed using an interactive session with GPT-4o via Azure OpenAI Studio, without relying on APIs, model fine-tuning, or external training data. This approach emphasizes the straightforward application of readily available generative AI models.

Datasets

The study utilized data from the ArchEHR-QA 2025 shared task. This dataset comprises patient questions and associated EHR data from MIMIC-III. Clinical note excerpts are provided with pre-assigned sentence numbers for citation. A development set of 20 cases includes manual "relevance" labels (essential, supplementary, not-relevant) for sentences, used to refine the sentence selection strategy. The test set consists of 100 cases without these relevance labels.

Key Insights and Ideas

  • Generative AI for Clinical QA is Feasible: The study demonstrates that a single, powerful generative AI model like GPT-4o can be effectively used to build a functional clinical QA system without extensive custom development or fine-tuning.
  • Self-Evaluation for Prompt Engineering: Using the generative AI model itself to analyze and improve its own sentence selection logic proved to be an effective strategy for refining the system's performance.
  • Importance of Sentence Selection: The "aha moment" lies in the realization that a well-designed sentence selection strategy is critical for ensuring the factuality and relevance of AI-generated clinical answers. Simply feeding raw text to a LLM is insufficient; targeted evidence extraction is key.
  • Traceability and Grounding are Paramount: The emphasis on citing evidence directly from the EHR is not just a task requirement but a fundamental principle for building trust and ensuring safety in clinical AI applications.

Key Results

The UTSamuel system was evaluated on 100 test cases.

  • Factuality: The system achieved a Factuality score of 47.8, which was higher than the mean (47.7) and median (45.3) among 30 participants. This indicates a consistent use of relevant evidence sentences. Strict micro F1 scores were also strong.
  • Relevance: For Relevance, the system scored 27.8, slightly lower than the mean (31.8) and median (33.1).
  • Readability: The SARI score of 56.7 was higher than the mean and median, suggesting that the generated answers were readable and cleanly edited.
  • Metrics like ROUGE-Lsum, BLEU, BERTScore, and MEDCON were slightly lower than the mean/median. The authors attribute this to the system's focus on generating short, evidence-backed answers rather than exact matches to reference summaries.

Limitations and Future Work

Limitations: * Tight Time Constraints: The system was developed under significant time pressure, limiting the exploration of more advanced techniques. * Web Interface Reliance: Accessing GPT-4o through a web interface, while simple, limits reproducibility and scalability. Programmatic validation and versioning were difficult. * Single Model Approach: The system exclusively used a single generative AI model.

Future Work: * Hybrid Systems: Experiment with multiple generative AI models and collaborative learning approaches. * Advanced Sentence Classification: Develop more sophisticated methods for classifying evidence sentences. * API Integration: Utilize APIs for improved reproducibility, scalability, and programmatic validation. * Structured Evaluation: Explore more structured evaluation pipelines for clinical QA systems.

Summary updated

BibTeX
@inproceedings{reason2025utsamuel,
  author = {Samuel Reason and Liwei Wang and Hongfang Liu and Ming Huang},
  title = {{UTSamuel at ArchEHR-QA 2025: A Clinical Question Answering System for Responding to Patient Portal Messages Using Generative AI}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
}
Figure