PaniniQA: Enhancing Patient Education Through Interactive Question Answering

Pengshan Cai, Zonghai Yao, Fei Liu, Dakuo Wang, Meghan Reilly, Huixue Zhou, Lingxi Li, Yi Cao, Alok Kapoor, Adarsha Bajracharya, Dan Berlowitz, Hong Yu

TL;DR

PaniniQA is an interactive patient education system that uses Large Language Models (LLMs) and medical information extraction to help patients understand their hospital discharge instructions. By engaging patients in a "Dialogic Reading" sequence—asking questions and providing corrective feedback—the system significantly improves a patient's mastery of their medical care plan compared to simply reading a static document.

What Problem Does This Paper Address?

When patients are discharged from the hospital, they are often handed a complex "discharge instruction" document. Research shows that patients immediately forget up to 80% of the information provided, and many struggle with low health literacy. This "discharge gap" leads to poor self-care at home, medication errors, and high readmission rates.

While patient portals allow digital access to these records, they remain passive. The authors argue that patient education needs to move from passive reading to active engagement. The challenge lies in automatically generating questions that are clinically relevant, grounded strictly in the patient's specific instructions (avoiding "hallucinations" or overly technical medical trivia), and providing helpful feedback when a patient answers incorrectly.

Methods

The researchers developed PaniniQA, which is inspired by the "Dialogic Reading" educational theory. It follows the PEER sequence: Prompt the user, Evaluate their answer, Expand on the information, and Repeat.

The system consists of three main technical components:

  1. Information Extraction (IE): To ensure questions are relevant, the authors fine-tuned a ClinicalRoBERTa model to identify 11 types of medical events (e.g., symptoms, medications, tests) and 6 types of relations (e.g., "Test result" or "Symptom caused by Disease") within the text.
  2. Question Generation (QG):
    • Vanilla GPT-3.5: A baseline that generates questions directly from the text.
  3. Enhanced (GPT+IE): Uses the extracted medical events as "triggers" to force the LLM to ask about specific, salient clinical points.
  4. Answer Verification: Using a chat-optimized LLM (ChatGPT), the system compares the patient's response to the discharge notes. It doesn't just say "Right" or "Wrong"; it provides a "physician-like" explanation to correct misunderstandings.

Datasets

  • MIMIC-III: The primary source of raw clinical data, consisting of de-identified health records from the Beth Israel Deaconess Medical Center.
  • PaniniQA Dataset: The authors created a new benchmark by having medical experts annotate 458 discharge notes from MIMIC-III for medical events and relations.
  • Synthesized Instructions: For human evaluation, they used 30 synthesized discharge instructions to ensure data privacy while maintaining clinical realism.

Key Insights and Ideas

  • The "Scope" Problem: A major finding was that standard LLMs often ask "out-of-scope" questions. For example, if a patient had a heart attack, a vanilla LLM might ask, "How does a cardiac catheterization work?" This requires external medical knowledge. PaniniQA’s IE-grounding ensures the system asks, "What was the cause of your chest pain?", which is answerable from the text and more educationally effective.
  • Active Learning via Cloze Tests: To measure if patients actually learned, the authors used "Cloze Tests" (fill-in-the-blank). They found that the act of answering a question—even if the AI wasn't perfect—forced patients to process the information more deeply than just reading.
  • Feedback is the "Aha" Moment: Simply asking a question (Condition Q) improved scores, but providing corrective feedback (Condition QA) led to the highest retention. This confirms that the "Evaluate and Expand" steps of the PEER sequence are critical for medical education.

Key Results

  • Learning Outcomes: In human trials, patients who just read the instructions scored 70.4% on comprehension tests. Those using the interactive PaniniQA (GPT+IE) scored 88.3%, approaching the performance of human-written questions (92.7%).
  • IE Accuracy: The ClinicalRoBERTa model achieved an F1-score of 78.07% for identifying medical relations, proving that specialized models still outperform general ones in niche clinical extraction tasks.
  • Preference: Human evaluators (both physicians and "patients") consistently ranked the IE-enhanced questions higher in "Appropriateness" and "Education Outcome" compared to questions generated by vanilla GPT-3.5.

Limitations and Future Work

  • Language Fluency: Some AI-generated questions were noted to be slightly redundant or lacked the natural "flow" of a human doctor.
  • Health Literacy Adaptation: The current system does not yet adjust the complexity of its language based on the individual patient’s education level or cognitive load.
  • Clinical Scope: The study focused on ICU data from MIMIC-III. The authors note the system needs to be tested in other contexts, such as surgical recovery or chronic disease management (e.g., cancer treatment).
  • Bias: Like all LLM-based systems, there is a risk of the model inheriting biases from its training data regarding specific demographic groups or medical conditions.

Summary updated

Abstract

Patient portal allows discharged patients to access their personalized discharge instructions in electronic health records (EHRs). However, many patients have difficulty understanding or memorizing their discharge instructions. In this paper, we present PaniniQA, a patient-centric interactive question answering system designed to help patients understand their discharge instructions. PaniniQA first identifies important clinical content from patients' discharge instructions and then formulates patient-specific educational questions. In addition, PaniniQA is also equipped with answer verification functionality to provide timely feedback to correct patients' misunderstandings. Our comprehensive automatic and human evaluation results demonstrate our PaniniQA is capable of improving patients' mastery of their medical instructions through effective interactions
BibTeX
@article{cai2023paniniqa,
  author = {Pengshan Cai and Zonghai Yao and Fei Liu and Dakuo Wang and Meghan Reilly and Huixue Zhou and Lingxi Li and Yi Cao and Alok Kapoor and Adarsha Bajracharya and Dan Berlowitz and Hong Yu},
  title = {{PaniniQA: Enhancing Patient Education Through Interactive Question Answering}},
  journal = {Trans. Assoc. Comput. Linguistics},
  year = {2023},
  doi = {10.1162/tacl\_a\_00616},
}
Figure