KnowLab at RadSum23: comparing pre-trained language models in radiology report summarization

Jinge Wu, Daqian Shi, Abul Hasan, Honghan WuACL 2023

TL;DR

This paper tackles the challenge of automatically summarizing radiology reports, a crucial task for efficient medical information retrieval. The authors explore fine-tuning large pre-trained language models (BART and T5) using a novel two-stage approach on two large radiology report datasets, demonstrating competitive performance on the RadSum23 shared task.

What Problem Does This Paper Address?

Radiology reports are dense documents filled with specialized medical terminology and often follow a specific structure. Extracting key findings and conclusions efficiently is vital for clinicians to quickly grasp patient status and make informed decisions, especially in time-sensitive medical environments. Traditional summarization methods struggle with this domain-specific language and complexity. While large language models (LLMs) have shown promise in general text summarization, adapting them effectively to the nuances of medical reports, particularly radiology reports, remains an active area of research. This paper aims to bridge this gap by investigating how state-of-the-art generative LLMs can be fine-tuned to produce high-quality, accurate summaries of radiology reports.

Methods

The core of this research lies in a two-stage fine-tuning approach applied to pre-trained generative language models. The authors specifically leverage BART (a denoising auto-encoder for sequence-to-sequence tasks) and T5 (a text-to-text transformer).

The two-stage process involves:

  1. Full fine-tuning on the larger MIMIC-III dataset to adapt the models to the general domain of radiology reports.
  2. A second round of fine-tuning on the MIMIC-CXR dataset, where the last two layers of the encoder and decoder are frozen. This "freezing" technique is hypothesized to prevent overfitting to the specific training data and promote better generalization to unseen data.

This approach aims to combine the broad knowledge of pre-trained models with domain-specific adaptation, leveraging the strengths of different datasets.

Datasets

The study utilizes two primary datasets for training and evaluation:

  • MIMIC-III: A large dataset containing free-text radiology reports from various anatomical regions (head, abdomen, chest, etc.) from the Beth Israel Deaconess Medical Center.
  • MIMIC-CXR: A subset of MIMIC-IV, specifically focusing on chest radiology reports.

The RadSum23 shared task organizers provided pre-defined train, validation, and test splits for these datasets.

Key Insights and Ideas

  • Two-stage fine-tuning is effective: The paper demonstrates that a sequential fine-tuning process, especially with layer freezing in the second stage, can lead to improved performance compared to single-stage fine-tuning. This suggests a benefit in gradually adapting models to domain-specific tasks.
  • Model strengths vary: Different models (BART and T5) and their variants exhibit varying performance across different evaluation metrics and datasets. This highlights that there isn't a single "best" model for all summarization scenarios, and the choice might depend on the specific requirements of the task and the evaluation criteria.
  • Domain-specific models show promise: While not the primary focus, the inclusion of models like BioBART and SciFive (pre-trained on biomedical corpora) provides a baseline for comparison, indicating the potential benefits of domain-specific pre-training.

Key Results

The paper reports experimental results on both MIMIC-III and MIMIC-CXR datasets using metrics like BLEU, ROUGE, Bertscore, CheXbert, and RadGraph.

  • MIMIC-III: The BART base model generally achieved high overall performance. BioBART showed strong performance on Bertscore and RadGraph.
  • MIMIC-CXR: The T5 model achieved the best performance on the MIMIC-CXR test set. The two-stage fine-tuning approach, particularly with layer freezing, generally yielded better results than a single-stage baseline. Notably, the authors' approach was ranked third on the RadSum23 MIMIC-CXR hidden test set leaderboard.

Specific scores are detailed in Tables 3 and 4, showing ranges for BLEU4 from ~11-23, RougeL from ~27-47, Bertscore from ~47-67, and F1 scores (CheXbert/RadGraph) from ~29-75, depending on the model, dataset, and test split.

Limitations and Future Work

The authors acknowledge several limitations and suggest future directions:

  • Limitations of current training: The current fine-tuning process has limitations, and further optimization is possible.
  • Exploring alternative strategies: The authors plan to investigate alternative training strategies and hyper-parameter tuning methods to further enhance model performance.
  • Larger datasets and broader scope: They recognize the importance of testing on larger datasets and expanding evaluations to include other natural language processing tasks.
  • Understanding layer freezing: The finding that freezing the last two layers can be advantageous requires further investigation to fully understand its effects.
  • Architecture design impact: There's a need for better understanding of how architecture design choices influence performance and how to optimally fine-tune accordingly for automatic clinical summarization.

Summary updated

BibTeX
@inproceedings{wu2023knowlab,
  author = {Jinge Wu and Daqian Shi and Abul Hasan and Honghan Wu},
  title = {{KnowLab at RadSum23: comparing pre-trained language models in radiology report summarization}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure