nav-nlp at RadSum23: Abstractive Summarization of Radiology Reports using BART Finetuning

Sri Macharla, Ashok Madamanchi, Nikhilesh KancharlaACL 2023

TL;DR

This paper tackles the challenge of automatically generating concise summaries of radiology reports, a crucial but time-consuming task for medical professionals. By fine-tuning pre-trained transformer models like BART and T5 on the MIMIC-III dataset, the researchers achieved promising results in abstractive summarization, demonstrating the potential of these advanced NLP techniques in healthcare.

What Problem Does This Paper Address?

Radiology reports are essential for documenting medical imaging examinations, but they can be lengthy and complex. Medical professionals often need to extract key findings and conclusions to make clinical decisions. This manual summarization process is time-consuming and inefficient, leading to potential delays and increased workload. This paper addresses the need for an automated solution to generate accurate and concise summaries of the "impression" section of radiology reports directly from the "findings" section, thereby improving workflow efficiency and clinical communication.

Methods

The researchers employed a transfer learning approach, leveraging pre-trained transformer models. Specifically, they fine-tuned two prominent encoder-decoder architectures: BART (Bidirectional and Auto-Regressive Transformers) and T5 (Text-to-Text Transfer Transformer). These models were trained on the MIMIC-III dataset, a large collection of de-identified clinical data. The fine-tuning process aimed to generate abstractive summaries, meaning the models learned to produce new sentences that capture the essence of the original text, rather than just extracting existing sentences. The Hugging Face Transformers library was used for implementing and fine-tuning these models.

Datasets

The primary dataset used in this research is the MIMIC-III database. This publicly available clinical database contains de-identified health data for over 50,000 patients, including radiology reports from various modalities (CT Head, CT Abdomen, CT Chest, MR Head, CT Spine, CT Neck) and anatomies. The dataset was divided into training, validation, and testing subsets. The researchers also mention using the BioNLP 2023 workshop's Task 1B dataset, which is based on MIMIC-III, for evaluation.

Key Insights and Ideas

  • Transformer models are highly effective for medical text summarization: The study demonstrates that fine-tuning large pre-trained transformer models like BART can yield significant improvements in abstractive summarization of complex medical reports.
  • Abstractive summarization is crucial for clinical efficiency: The ability to generate novel, concise summaries rather than just extracting sentences is key to saving valuable time for medical professionals.
  • BART outperforms T5 in this specific task: While both models showed promise, the results suggest that BART, particularly the BART-base model, achieved better performance on the radiology report summarization task.

Key Results

The researchers evaluated their models using standard metrics such as ROUGE-L, BertScore, and F1-RadGraph. Their best-performing system, a fine-tuned BART-base model, achieved the following scores on the MIMIC-III open-test set:

  • ROUGE-L: 32.33
  • BertScore: 54.49
  • F1-RadGraph: 32.68

These scores are competitive within the context of the BioNLP 2023 shared task, as indicated by their ranking in Table 3.

Limitations and Future Work

The paper acknowledges that resource constraints led them to use BART-base and T5-base models, suggesting that larger or more specialized models might yield even better results. The authors also mention that the evaluation was performed on specific test sets (open-test and hidden-test), and further validation on diverse real-world clinical data could be beneficial. Future work could explore incorporating multi-modal information (e.g., images) into the summarization process and investigating more advanced techniques for ensuring the factual accuracy and clinical relevance of the generated summaries.

Summary updated

BibTeX
@inproceedings{macharla2023nav,
  author = {Sri Macharla and Ashok Madamanchi and Nikhilesh Kancharla},
  title = {{nav-nlp at RadSum23: Abstractive Summarization of Radiology Reports using BART Finetuning}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure