Abstract
Discharge communication is a critical yet underexplored component of patient care, where the goal shifts from diagnosis to education. While recent large language model (LLM) benchmarks emphasize in-visit diagnostic reasoning, they fail to evaluate models' ability to support patients after the visit. We introduce DischargeSim, a novel benchmark that evaluates LLMs on their ability to act as personalized discharge educators. DischargeSim simulates post-visit, multi-turn conversations between LLM-driven DoctorAgents and PatientAgents with diverse psychosocial profiles (e.g., health literacy, education, emotion). Interactions are structured across six clinically grounded discharge topics and assessed along three axes: (1) dialogue quality via automatic and LLM-as-judge evaluation, (2) personalized document generation including free-text summaries and structured AHRQ checklists, and (3) patient comprehension through a downstream multiple-choice exam. Experiments across 18 LLMs reveal significant gaps in discharge education capability, with performance varying widely across patient profiles. Notably, model size does not always yield better education outcomes, highlighting trade-offs in strategy use and content prioritization. DischargeSim offers a first step toward benchmarking LLMs in post-visit clinical education and promoting equitable, personalized patient support.
Introduction
The transition from hospital to home is a recognized "danger zone" in healthcare. Studies indicate that between 40% and 80% of patients forget or misunderstand their discharge instructions almost immediately after leaving the hospital, leading to medication errors, missed follow-ups, and preventable readmissions. While the AI research community has produced several benchmarks for clinical reasoning, these focus almost entirely on diagnosis—testing if a model can identify a condition. They largely ignore the educational phase: whether the model can explain a care plan so a patient actually understands it. DischargeSim addresses this gap by providing a dynamic, multi-turn simulation framework for post-visit counseling.
Methods and Framework
DischargeSim utilizes a multi-agent simulation involving two primary agents: a DoctorAgent (the LLM under evaluation) and a PatientAgent (a simulated patient with a specific psychosocial profile). To ensure clinical relevance, the researchers sampled 49 gold-standard discharge notes from the MIMIC-IV database to serve as the ground truth for simulations.
The interaction is structured across six clinically validated stages: Diagnosis, Tests/Treatments, Return-to-Hospital Indicators, Medication, Post-discharge Treatment, and Follow-up. The evaluation is uniquely thorough, assessing dialogue quality via LLM-as-judge (DeepSeek-V3), the accuracy of generated AHRQ-compliant checklists, and, most importantly, patient comprehension. After the simulation, the PatientAgent takes a 10-question multiple-choice exam to determine how much information was actually retained.
Key Results and Insights
The benchmarking of 18 LLMs, including the GPT, LLaMA, and Qwen series, revealed several critical trends in AI-driven medical education.
The "Fluency vs. Function" Gap
One of the most significant findings is that larger models produce highly fluent and empathetic responses, but this does not always translate to better patient comprehension. A model may sound professional while failing to mention a critical medication dosage. For instance, while GPT-5-mini and LLaMA-3.1-70B led in patient comprehension (84.89% and 84.29% exam accuracy, respectively), the scaling trend for comprehension is much noisier than the trend for language delivery.
Profile Sensitivity and Equity
Models perform significantly better when interacting with "high-literacy" patients. When the PatientAgent is initialized as "low-literacy" or "anxious," the performance of the DoctorAgent drops. This highlights a risk that AI could exacerbate existing healthcare inequities if not specifically tuned for diverse patient needs. Furthermore, some models were found to "over-index" on communication strategies—such as asking open-ended questions—at the expense of delivering necessary medical information.
Limitations and Future Work
While DischargeSim provides a robust starting point, the authors acknowledge several constraints. The benchmark is currently limited to English-language scenarios and Western medical contexts. Additionally, while the PatientAgents are dynamic, they rely on LLM-generated responses which may not fully capture the unpredictability of a real human patient. Future work will look toward incorporating "human-in-the-loop" evaluations and expanding the benchmark to specialized fields like pediatrics and oncology, where discharge communication is uniquely complex.
Summary updated