TL;DR
This paper presents BDKG's winning system for the MEDIQA 2021 Radiology Report Summarization task. They achieved top performance by fine-tuning the PEGASUS model, incorporating domain adaptation for generalization, and using ensemble and normalization techniques. This work highlights the effectiveness of pre-trained models combined with targeted adaptation strategies for complex medical text summarization.
What Problem Does This Paper Address?
Radiology reports are crucial for patient care, but manually summarizing the detailed findings into concise impressions is time-consuming and prone to errors. The MEDIQA 2021 task specifically focused on automatically generating these impressions from the "findings" and "background" sections of radiology reports. A key challenge highlighted in this task is the need for systems to generalize and transfer their knowledge to reports from different hospitals or data sources, as training and evaluation datasets often come from distinct origins (e.g., MIMIC-CXR vs. Indiana chest X-ray collection). This discrepancy poses a significant hurdle for models trained on one distribution to perform well on another.
Methods
The BDKG system builds upon a Transformer-based encoder-decoder architecture, specifically leveraging the pre-trained PEGASUS model, which is designed for abstractive text summarization. Their approach involves several key modules:
- Transformer Encoder-Decoder Training: The core of the system is a PEGASUS model fine-tuned on the MIMIC-CXR training dataset. The input to the model is a concatenation of the "findings" and "background" sections, with a special token separating them, and the target output is the "impression" section.
- Domain Adaptation: To address the generalization challenge, a domain adaptation module is employed. This involves further fine-tuning the model on a small amount of data that matches the source of the test set. For the Indiana test split, they used the provided validation set. For the Stanford test split, they used a subset of the training data that was retrieved using ElasticSearch to find reports with similar findings.
- Model Ensemble: To improve robustness, an ensemble module combines predictions from multiple models. They trained several models with different random seeds and aggregated their predictions, selecting the one with the highest overall similarity to the others.
- Negative Impression Normalization: A final heuristic normalization module is used to standardize negative impressions. Specifically, any impression starting with "No acute" is normalized to "No acute cardiopulmonary abnormality," which was identified as the most frequent negative impression in the validation sets. This is particularly useful for the Stanford test split, which lacks a corresponding training/validation organization.
Datasets
The task utilized official data splits from different sources:
- Training Split: 91,544 chest radiology reports from the MIMIC-CXR database.
- Validation Split I: 2,000 reports from MIMIC-CXR (same distribution as training).
- Validation Split II: 2,000 reports from the Indiana chest X-ray collection.
- Test Split I: 300 reports from the Indiana chest X-ray collection.
- Test Split II: 300 reports from the Stanford Hospital collection.
The key challenge here is that the test splits (Indiana and Stanford) originate from different sources than the training split (MIMIC-CXR), necessitating robust generalization.
Key Insights and Ideas
- Pre-trained models are powerful, but domain adaptation is crucial: While PEGASUS provided a strong baseline, the significant performance boost on the Indiana test split after domain adaptation highlights the necessity of adapting models to specific data distributions, especially in specialized domains like medical imaging.
- Simple heuristics can be effective: The negative impression normalization strategy, though simple, proved beneficial, particularly for the Stanford test split. This suggests that for certain predictable patterns, rule-based approaches can complement complex deep learning models.
- Ensembling adds value: The ensemble method consistently improved performance across various models and datasets, demonstrating its utility in stabilizing predictions and improving overall accuracy.
- Generalization is the main challenge: The task's design, with distinct training and testing data sources, directly targets the ability of models to perform well on unseen data distributions, a critical aspect for real-world deployment.
Key Results
The BDKG system achieved the 1st place in the MEDIQA 2021 Radiology Report Summarization task, with an overall ROUGE-2 score of 0.436 on the entire test set. They consistently outperformed other systems across all test splits and metrics. The paper also shows that domain adaptation significantly boosted performance on the Indiana test split (from ROUGE-2 of 0.3062 to 0.5956), and negative impression normalization provided gains on the Stanford test split (from 0.2486 to 0.2769).
Limitations and Future Work
The authors acknowledge that the domain adaptation strategy for the Stanford test split, which involved retrieving similar reports from the training set, did not improve performance and was ultimately discarded in favor of using the model trained directly on MIMIC-CXR. This suggests that the retrieval-based adaptation might not have been optimal or that the Stanford data was too dissimilar. Future work could explore more advanced domain adaptation techniques or investigate why the initial retrieval-based approach was less effective. The paper also implicitly suggests that further exploration of different pre-trained models and their fine-tuning strategies could yield additional improvements.
Summary updated