Personalizing Prostate Cancer Education for Patients Using an EHR-Integrated LLM Agent

Yuexing Hao, Jason Holmes, Mark R. Waddle, Nathan Y. Yu, Kirstin Vickers, Heather R. Preston, Drew Margolin, Corinna E. Löckenhoff, Aditya Vashistha, Marzyeh Ghassemi, Shakiba Kalantari, Wei Liu, Liu, Wei

Abstract

Cancer patients often lack timely education and personalized support due to clinician workload. This quality improvement study develops and evaluates a Large Language Model (LLM) agent, MedEduChat, which is integrated with the clinic's electronic health records (EHR) and designed to enhance prostate cancer patient education. Fifteen non-metastatic prostate cancer patients and three clinicians recruited from the Mayo Clinic interacted with the agent between May 2024 and April 2025. Findings showed that MedEduChat has a high usability score (UMUX 83.7 out of 100) and improves patients' health confidence (Health Confidence Score rose from 9.9 to 13.9). Clinicians evaluated the patient-chat interaction history and rated MedEduChat as highly correct (2.9 out of 3), complete (2.7 out of 3), and safe (2.7 out of 3), with moderate personalization (2.3 out of 3). This study highlights the potential of LLM agents to improve patient engagement and health education.

Introduction

The "transition to treatment" phase for cancer patients is often characterized by significant anxiety and information gaps. While patients require detailed, accessible guidance, clinicians are frequently overburdened, limiting their ability to provide personalized support. Existing solutions, such as generic brochures or standard chatbots, often fail because they provide static information without the specific context of a patient’s unique diagnosis and treatment history. This study introduces MedEduChat, an LLM-powered agent integrated directly into Electronic Health Records (EHR) to provide personalized, high-fidelity education for prostate cancer patients.

Methods

MedEduChat was developed using GPT-4o via a HIPAA-compliant Azure endpoint and integrated with the Mayo Clinic’s EHR systems (Epic and Aria). The system architecture prioritizes clinical precision and safety through several key mechanisms:

  • Educational Framework: The agent follows the "5E Model" (Engage, Explore, Explain, Elaborate, and Evaluate).
  • Deterministic Retrieval: To ensure accuracy, the team utilized rule-based keyword retrieval (SQL) instead of standard vector-based Retrieval-Augmented Generation (RAG), ensuring data is explicitly linked to verified medical terms.
  • Safety Infrastructure: A multi-stage workflow manages risks, including pre-filtering prompts, post-generation auditing for hallucinations, and an escalation protocol for high-risk topics like medication dosing or emotional distress.
MedEduChat system architecture and safety escalation workflow.

The study utilized real-time internal EHR data (radiotherapy plans, clinical notes, etc.) and external knowledge bases like NCCN guidelines. Evaluation involved 85 pairs of anonymized patient-agent interactions graded by three independent clinical professionals.

Results and Insights

The pilot study yielded strong quantitative and qualitative results. Patients reported that the AI acted as a helpful buffer, easing anxiety while waiting for clinician responses. Notably, the agent helped patients "unlearn" misconceptions by providing specific data regarding their treatment plans.

Clinician evaluations confirmed the high quality of the agent's responses across several dimensions, including correctness and safety.

Clinician ratings of MedEduChat performance across key metrics.

Key performance metrics included:

  • Usability: A UMUX score of 83.7/100.
  • Health Confidence: Patient scores improved from 9.9 to 13.9 (out of 16).
  • Clinical Accuracy: Rated 2.9/3.0 for correctness and 2.7/3.0 for safety.
  • Personalization: Rated 2.3/3.0, indicating effective use of patient data with room for further tailoring to individual literacy levels.

Discussion and Limitations

While promising, the study has limitations. The pilot participants were primarily older, white, and highly educated, necessitating further testing with more diverse populations. Additionally, the current text-only interface cannot interpret medical images or audio. The integrity of the LLM's output remains dependent on the quality of the underlying EHR data. Future work includes a preregistered randomized controlled trial (RCT) to compare MedEduChat against standard educational materials.

Summary updated

Abstract

Cancer patients often lack timely education and personalized support due to clinician workload. This quality improvement study develops and evaluates a Large Language Model (LLM) agent, MedEduChat, which is integrated with the clinic's electronic health records (EHR) and designed to enhance prostate cancer patient education. Fifteen non-metastatic prostate cancer patients and three clinicians recruited from the Mayo Clinic interacted with the agent between May 2024 and April 2025. Findings showed that MedEduChat has a high usability score (UMUX 83.7 out of 100) and improves patients' health confidence (Health Confidence Score rose from 9.9 to 13.9). Clinicians evaluated the patient-chat interaction history and rated MedEduChat as highly correct (2.9 out of 3), complete (2.7 out of 3), and safe (2.7 out of 3), with moderate personalization (2.3 out of 3). This study highlights the potential of LLM agents to improve patient engagement and health education.
BibTeX
@article{hao2025personalizing,
  author = {Yuexing Hao and Jason Holmes and Mark R. Waddle and Nathan Y. Yu and Kirstin Vickers and Heather R. Preston and Drew Margolin and Corinna E. Löckenhoff and Aditya Vashistha and Marzyeh Ghassemi and Shakiba Kalantari and Wei Liu and Liu, Wei},
  title = {{Personalizing Prostate Cancer Education for Patients Using an EHR-Integrated LLM Agent}},
  journal = {Npj Digit. Medicine},
  year = {2025},
  doi = {10.1038/s41746-025-02166-0},
}
Figure