TL;DR
This paper explores the use of the BART language model to automatically generate discharge summaries from Swedish clinical notes. While previous research has focused on other languages, this is the first attempt for Swedish. Despite fine-tuning BART on a large dataset, the generated summaries were found to be of low quality, lacking important clinical information and sometimes containing hallucinations, indicating that current models are not yet ready for real-world clinical deployment.
What Problem Does This Paper Address?
Discharge summaries are crucial documents that summarize a patient's hospital stay, facilitating communication between healthcare providers. However, manual creation is time-consuming and often results in summaries that are delayed, incomplete, or of poor quality. Text summarization, particularly using advanced Transformer models like BART, offers a potential solution to automate this process. While this has been explored in English and other languages, there's a significant gap in research for Swedish clinical text, which this paper aims to fill.
Methods
The researchers fine-tuned a pre-trained BART language model (KB-BART) on Swedish clinical data. The core of their methodology involved:
- Data Pre-processing: Since the raw data lacked explicit links between patient notes and discharge summaries, they developed a method to group notes into "care episodes" chronologically and pair them with their corresponding discharge summaries. Pairs where the discharge summary was longer than the care episode text were discarded.
- Data Subsetting: To investigate the impact of data quality, they created four subsets of the paired data:
- FULL: All pairs.
- FILT1: Pairs where at least one term from the care episode appeared in the discharge summary.
- FILT2: Pairs where at least 15% of terms from the care episode appeared in the discharge summary.
- METR: Pairs filtered using metrics for semantic coherence, topic similarity, and redundancy.
- Model Training: Four separate KB-BART models were fine-tuned, one for each data subset.
- Evaluation: A two-pronged evaluation approach was used:
- Automatic Evaluation: Using ROUGE scores (ROUGE-1, ROUGE-2, ROUGE-L, ROUGE-S) to compare generated summaries against benchmarks like "Oracle" (ideal extractive summary) and "Random" (randomly selected sentences).
- Manual Evaluation: A retired senior nurse and clinical coder assessed ten randomly selected summaries from the best-performing model against 12 criteria, including the presence of key information, potential hallucinations, and readability.
Datasets
- Training and Evaluation Data: The study utilized the "Stockholm EPR Gastro ICD-10 Pseudo Corpus II" (Corpus II) data set from the Swedish Health Record Research Bank (Health Bank). This corpus contains over 350,000 patient notes from the Karolinska University Hospital, Sweden, collected between 2006 and 2014. The data was automatically de-identified and anonymized.
- Benchmarks:
- Oracle: An extractive summarization benchmark designed to achieve high ROUGE scores by selecting sentences that best match an existing summary.
- Random: An extractive summarization benchmark that randomly combines sentences.
Key Insights and Ideas
- Swedish Clinical Text is a New Frontier: This paper highlights the lack of research in automatically generating discharge summaries for Swedish clinical text, underscoring the need for language-specific NLP models.
- Data Quality Matters, But Not Enough: While filtering data (especially the FILT2 subset) led to the best automatic ROUGE scores, the overall quality of generated summaries remained low. This suggests that simply improving data filtering might not be sufficient to overcome fundamental challenges.
- ROUGE Scores Don't Tell the Whole Story: The best model achieved decent ROUGE scores compared to benchmarks, but manual evaluation revealed significant shortcomings in clinical relevance and accuracy. This emphasizes the importance of human evaluation for critical applications like healthcare.
- Hallucinations are a Major Concern: The manual evaluation identified severe hallucinations in the generated summaries, a critical issue that prevents their use in a clinical setting where accuracy is paramount.
- Clinical Language Generation is Feasible: The study noted the ease with which the BART model could generate clinical language, suggesting potential for using such models to generate synthetic data for future training.
Key Results
- Automatic Evaluation: The model fine-tuned on the FILT2 subset performed best among the four models, outperforming the Random benchmark but not the Oracle benchmark. However, the margin of improvement over Random was not significant.
- Manual Evaluation: The best-performing model (based on FILT2) produced summaries that were prone to:
- Omitting clinically important information: Especially reason for admission and long-term diagnoses.
- Generating severe hallucinations: Information not present in the source notes.
- Poor readability: Lacking flow and overall coherence.
- Overall Conclusion: The best model produced discharge summaries with "severe shortcomings" and is "far from ready to be used in any real-life clinical setting."
Limitations and Future Work
- Data Limitations: The study acknowledges that the data set had no explicit structure for grouping patient notes into care episodes or linking them to discharge summaries. This pairing was largely based on assumptions, meaning the accuracy of these pairings is not guaranteed.
- Model Performance: The generated summaries were deemed low quality, lacking clinical relevance and containing hallucinations.
- Future Work Suggestions:
- Improved Data Pairing: Develop more robust methods for accurately grouping patient notes into care episodes and pairing them with discharge summaries.
- Explore Other Summarization Methods: Investigate Extractive Text Summarization (ETS) and Hybrid Text Summarization (HTS), as HTS has shown promise in previous research.
- Domain-Specific Pre-training: Replace the general Swedish pre-trained BART model with one pre-trained specifically on Swedish clinical text to potentially improve performance.
- Ethical Considerations: Address ethical issues related to generating text containing personal information, especially concerning GDPR.
Summary updated