TL;DR
This paper introduces a novel domain-adaptive pre-training strategy for instruction-tuned Large Language Models (LLMs) to improve their ability to generate radiology report impressions. By further pre-training existing LLMs on a large corpus of radiology reports, the authors significantly enhance their performance in summarizing medical findings, even outperforming standard fine-tuning methods in a zero-shot setting.
What Problem Does This Paper Address?
Radiology reports are crucial for communicating medical image interpretations. They typically consist of "FINDINGS" (what's observed) and "IMPRESSIONS" (a summary of key findings and recommendations). While general-purpose LLMs like ChatGPT show impressive generalization, they struggle with the nuances of medical text, often producing impressions that are too verbose or incomplete. This is largely due to insufficient exposure to specialized medical data during their initial training. Existing domain-specific LLMs (like BioBERT) also have limitations, often being trained on smaller, less diverse datasets. The challenge is to adapt these powerful LLMs to accurately and concisely summarize complex medical information, bridging the gap between general language understanding and specialized medical reporting.
Methods
The core contribution is a new domain adaptation paradigm called "general-pretrain-prompt-tune-and-special-pretrain." This approach builds upon existing instruction-tuned LLMs (specifically, Bloomz-7b1). Here's how it works:
- General Pretraining & Prompt-Tuning: This is the foundation of instruction-tuned LLMs, where they learn general language patterns and how to follow instructions across various tasks.
- Domain-Adaptive Pretraining (DAPT): The key innovation. The instruction-tuned LLM is further pre-trained using a causal language modeling objective on a large dataset of radiology reports (MIMIC-IV). This step imbues the model with domain-specific knowledge.
- Fine-tuning: The model is then fine-tuned on the specific task of radiology report summarization, using FINDINGS as input and aiming to generate IMPRESSIONS. To prevent catastrophic forgetting of the learned domain knowledge, only the last layer of the model is modified.
This strategy aims to leverage the strengths of both general LLMs and domain-specific adaptation, with the added benefit of maintaining the instruction-following capabilities.
Datasets
- Pretraining: MIMIC-IV radiology reports dataset. This is a large dataset containing over 2.3 million radiology reports. For their experiments, they utilized approximately 1.4 million reports, totaling 190 million tokens.
- Fine-tuning & Evaluation: Datasets from the BioNLP 2023 workshop Task 1B: Radiology Report Summarization. This includes:
- MIMIC-III
- MIMIC-CXR (which includes images, though only text was used for this task)
- CHEXPERT
Key Insights and Ideas
- Specialist Pretraining is Key: The paper strongly suggests that simply finetuning a general LLM isn't enough for specialized domains like radiology. Further pre-training on domain-specific text (like radiology reports) is crucial for achieving high performance.
- Instruction Tuning + Domain Adaptation = Synergy: Combining the instruction-following capabilities of models like Bloomz with domain-specific pre-training (their "general-pretrain-prompt-tune-and-special-pretrain" approach) yields superior results compared to standard "pretrain-and-finetune" methods, especially in a zero-shot setting.
- Zero-Shot Power: The proposed method achieves strong performance even without task-specific fine-tuning on the target dataset, highlighting the effectiveness of the domain adaptation.
- Nuance of Medical Text: The error analysis reveals that standard n-gram overlap metrics (like ROUGE) can be insufficient for evaluating medical summaries. Semantically similar but not identical phrases can lead to low scores, and models can sometimes "hallucinate" or miss crucial context, underscoring the complexity of the task.
Key Results
- Ranked 1st: The proposed system (shs-nlp team) ranked 1st in the hidden-test category for Task 1B: Radiology Report Summarization at the BioNLP 2023 workshop.
- Superior to Pretrain-and-Finetune: In a zero-shot setting, their domain-adaptive pre-trained model (RadBloomz) demonstrated better performance than standard pretrain-and-finetune methods.
- MIMIC-III Performance: RadBloomz ranked first among participating systems on the MIMIC-III hidden test dataset (which contains only reports).
- MIMIC-CXR Performance: It ranked fourth on the MIMIC-CXR dataset, which is multimodal, showing effectiveness even when the task involves more than just text.
Limitations and Future Work
- English-Centric: The domain adaptation was performed on English reports only, limiting its direct applicability to multilingual settings.
- Data Overlap: The use of MIMIC-IV for pretraining might include overlapping data with MIMIC-III and MIMIC-CXR, posing a potential risk of information leak.
- Data Imbalance: The training data has an imbalance in represented anatomies and modalities, which might affect performance on underrepresented regions.
- Patient Diversity: The impact of patient diversity on model performance, especially for underrepresented communities, needs further study.
- Radiologist Variability: The paper acknowledges that radiologists have different reporting styles, and the consistency, uncertainty, or information richness of reports hasn't been studied in this context.
- Computational Throughput: The model's space and time complexity might make it unsuitable for on-premise or edge deployments, suggesting future work in model quantization and optimization.
- Hallucinations: The error analysis identified instances of numerical hallucinations and incorrect concept generation, indicating areas for improvement. Future work could explore methods to mitigate these issues.
Summary updated