TL;DR
This paper introduces MEDSUM-ENT, a multi-stage approach for generating medically accurate summaries of patient-provider dialogues using large language models (LLMs) like GPT-3. By breaking down summarization into entity extraction and refinement steps, MEDSUM-ENT significantly outperforms naive summarization methods, offering a promising step towards automating clinical documentation.
What Problem Does This Paper Address?
Medical providers spend a significant amount of time summarizing patient visits. These summaries are crucial for clinical decision-making, patient handoffs, and patient reference. However, generating accurate and coherent summaries from complex, often informal patient language is challenging. Minor inaccuracies can have detrimental effects on patient care. Existing automated summarization methods have shown limited success, often struggling to precisely capture medical concepts and their affirmation status (e.g., present, absent, unknown). This paper aims to address this gap by developing a more robust and medically accurate summarization system.
Methods
The core of MEDSUM-ENT is a multi-stage approach that leverages GPT-3. Instead of a single-shot summarization, the process is broken down:
- Medical Entity Extraction: The system first identifies key medical entities (symptoms, conditions, etc.) and their affirmation status (present, absent, unknown) from the patient's initial message (Reason for Encounter - RFE) and subsequent dialogue turns. This extraction is guided by carefully crafted prompts and in-context examples.
- Entity Resolver: An additional step is included to refine the "unknown" entities, using information from later dialogue turns to potentially update their status. This helps to reduce errors in the final summary.
- Summarization: The extracted and refined medical entities are then used as additional input, alongside the dialogue itself, to generate the final summary. This "prompt chaining" approach allows the LLM to leverage structured medical information.
The authors also explore different strategies for selecting in-context examples for the LLM, comparing semantic similarity with random selection.
Datasets
The study uses a dataset of 100 clinical encounters from a telehealth platform, consisting of patient-provider chat dialogues and corresponding ground-truth summaries. These summaries were initially generated by GPT-3 and then manually edited by physicians for correctness.
Key Insights and Ideas
- Decomposition is Key: Breaking down the complex task of medical summarization into smaller, sequential steps (entity extraction, refinement, then summarization) significantly improves accuracy.
- Structured Information Boosts LLMs: Providing LLMs with extracted, structured medical entities as input, rather than just raw dialogue, leads to more medically accurate and clinically relevant summaries.
- Random Example Selection Wins: Surprisingly, randomly selecting in-context examples for the LLM performed better than selecting examples based on semantic similarity. This suggests that diversity in examples might be more crucial than direct topical relevance for this task.
- "Pertinent Unknowns" and "Medical History" are Crucial: The multi-stage approach showed particular improvement in accurately capturing "Pertinent Unknowns" and "Medical History" sections, which are vital for clinical decision-making.
Key Results
- Human Preference: In human evaluations, MEDSUM-ENT (5-shot) summaries were preferred over a naive zero-shot baseline 66% to 34% of the time.
- GPT-Driven Metrics: Quantitative metrics (GPT-Recall, GPT-Precision, GPT-F1) showed that MEDSUM-ENT consistently outperformed the baseline, especially in the "Pertinent Unknowns" and "Medical History" sections. The entity resolver specifically improved performance in the "Pertinent Unknowns" section.
- No Clear Benefit from Higher K-shot: Increasing the number of shots (k-shot configurations) for entity extraction did not consistently lead to better performance, suggesting diminishing returns for GPT-3 in this specific task.
- Qualitative Findings: Expert evaluators noted that while MEDSUM-ENT captured relevant information well, 28% of summaries contained incorrect information that could potentially harm patients if acted upon, highlighting the need for further refinement and trust-building.
Limitations and Future Work
The authors acknowledge several limitations:
- API Dependency: Experiments were conducted using OpenAI's GPT-3 API, which has computational and accessibility constraints.
- Limited Data: The study used a small, proprietary dataset of English-language dialogues, limiting generalizability.
- Evaluation Challenges: Summarization evaluation is inherently difficult, relying on a small number of human annotators and automatic metrics.
Future work could involve investigating more advanced refinement modules, iterative summarization techniques, and conducting broader expert human evaluation studies to build trust and facilitate the adoption of model-assisted summarization in clinical practice.
Summary updated