TL;DR
This paper presents the SINAI team's approach to automating radiology report summarization for the BioNLP workshop. They leveraged pre-trained language models, specifically a domain-adapted SciFive model, to achieve strong performance on the task of generating concise "Impression" sections from full radiology reports. Their best model secured fourth place in the competition, highlighting the benefit of domain-specific pre-training.
What Problem Does This Paper Address?
Radiology reports are crucial for patient care, but their length and complexity can sometimes obscure key findings. The "Impression" section, intended to summarize the most important information, is vital for clear communication between radiologists and referring physicians. However, ensuring the accuracy and conciseness of these summaries is challenging. The ACL 2023 BioNLP workshop's Task 1B aimed to address this by focusing on automatic radiology report summarization, specifically the generation of the Impression section. While previous work often focused on Chest X-rays due to available datasets, this task expanded to include seven different modalities and anatomies, requiring more generalized summarization capabilities. The challenge specifically emphasized maximizing the factual correctness of the generated summaries.
Methods
The SINAI team employed a sequence-to-sequence approach, a standard paradigm for text generation tasks. Their core strategy involved fine-tuning pre-trained transformer models. They experimented with two main models:
- flan-t5-base: A general-domain T5 model that had undergone instruction fine-tuning on a wide variety of tasks. This served as their baseline general-purpose model.
- SciFive-base-Pubmed_PMC: This model is a T5 variant specifically pre-trained on a large corpus of biomedical literature. The hypothesis was that this domain-specific pre-training would lead to better performance on radiology reports.
To optimize their models, they utilized hyperparameter optimization with the Optuna framework, searching for optimal learning rates and weight decay.
Additionally, they explored Proximal Policy Optimization (PPO), a reinforcement learning technique, to further enhance the factual correctness of the generated summaries. The idea was to use the F1RadGraph metric (which assesses factual accuracy) as a reward signal to guide the model.
Datasets
The primary dataset used for this task was provided by the ACL 2023 BioNLP shared task. This dataset contained 73,255 radiology reports for the mono-modal challenge, split into training (5,9317 reports), development (7,413), and test (6,526) sets. The official evaluation was conducted on a hidden test set of 6,531 samples. These reports covered 2 different types of radiology explorations (CT and MRI) across 7 different anatomical regions.
Key Insights and Ideas
- Domain-specific pre-training is beneficial: The SciFive model, pre-trained on biomedical literature, consistently outperformed the general-domain flan-t5 model, suggesting that specialized knowledge significantly aids in understanding and summarizing radiology reports.
- Reinforcement learning for factual correctness is challenging: Despite the theoretical promise of PPO for improving factual accuracy, their experiment did not yield positive results. This suggests that applying RL to this specific task, especially with shared model layers, presents significant hurdles.
- Sequence-to-sequence models are well-suited: The fundamental sequence-to-sequence architecture proved effective for the summarization task, forming a solid foundation for their experiments.
Key Results
The SINAI team's best-performing system, the fine-tuned SciFive-base-Pubmed_PMC model, achieved an F1RadGraph score of 33.96. This result placed them fourth among the challenge participants.
While the general-domain flan-t5 model also performed reasonably well (33.71 F1RadGraph), the SciFive model's slight edge highlights the advantage of domain adaptation.
The PPO experiment, unfortunately, did not improve performance. The F1RadGraph score for the PPO-tuned model was 32.48, which was lower than both the original flan-t5 and SciFive models.
Limitations and Future Work
The authors acknowledge that the limitations of their system are directly tied to the pre-trained models they used. Specifically, they note that flan-t5 was pre-trained on a large-scale text dataset and may not have been assessed for existing biases. This means the model could potentially generate inappropriate content or replicate biases present in the training data. They caution against using such systems without thorough evaluation for safety and fairness concerns.
For future work, the team plans to focus on improving the factual correctness of the output through RL techniques. They aim to overcome the challenges encountered in their initial PPO experiment to better address the problem of generating accurate summaries. They also mention that the issue might be related to the "sequence-to-sequence" nature of their approach or the T5 model itself, suggesting further architectural or methodological exploration.
Summary updated