The More, The Better? A Critical Study of Multimodal Context in Radiology Report Summarization

Mong Yuan Sim, Wei Emma Zhang, Xiang Dai, Biaoyan Fang, Sarbin Ranjitkar, Arjun Burlakoti, Jamie Taylor, Haojie ZhuangEMNLP 2025

TL;DR

This paper investigates whether radiology images are truly necessary for generating accurate radiology report summaries (IMPRESSION sections), finding that current multimodal models often fail to effectively utilize them. The study suggests that the FINDINGS section alone is frequently sufficient, and proposes methods to encourage better multimodal integration, while also highlighting the need to redefine multimodal tasks when one modality might be redundant.

What Problem Does This Paper Address?

Radiology reports are crucial for communication between radiologists and physicians, with the IMPRESSION section summarizing key findings. Traditionally, summarization models focused on generating this section from the FINDINGS text. More recently, researchers have explored incorporating radiology images as a multimodal input, assuming this would improve summary quality by providing richer information. However, the actual benefit of these images remains unclear. This paper critically examines this assumption, questioning whether current multimodal models genuinely leverage visual information or if the FINDINGS text is already so comprehensive that images become redundant. This is particularly important in a clinical setting where summary accuracy directly impacts patient care.

Methods

The authors employ a multi-pronged approach to dissect the role of radiology images:

  • Model Behavior Analysis: They train and evaluate several text-only and multimodal summarization models (including large vision-language models or LVLMs) on radiology report summarization tasks.
  • Image Masking Ablation: To assess image utilization, they mask the input radiology images during inference time for multimodal models. A significant performance drop after masking would indicate reliance on visual input.
  • Two-Stage Fine-Tuning: To actively encourage image utilization, they experiment with a two-stage fine-tuning strategy. This involves first training the projection layer to map visual features to the language model's embedding space, followed by training the language model.
  • Exclusive Set Construction: To disentangle model behavior from dataset characteristics, they create an "exclusive set" by removing sentences from the FINDINGS section that directly correspond to diagnoses also present in the image. This forces models to rely on the image for those specific diagnoses.
  • Expert Annotation Study: Medical experts with extensive experience were asked to determine if they could write the IMPRESSION section using only the FINDINGS text, without the radiology image.

Datasets

The study utilizes three publicly available datasets:

  • MIMIC-CXR: A large-scale dataset with over 377,000 chest X-rays and associated reports.
  • OpenI: A smaller dataset from Indiana University with 3,346 reports.
  • CheXpert: A large-scale chest radiograph dataset from Stanford Hospital.

The authors follow the official train/validation/test splits for MIMIC-CXR and OpenI. For CheXpert, they use a shared task version and sample from its training set.

Key Insights and Ideas

  • Redundancy is Common: The core insight is that for many radiology reports, the FINDINGS section already contains sufficient information to generate an accurate IMPRESSION, making the radiology image redundant for summarization.
  • Multimodal Models Don't Always "See": Current multimodal models often fail to effectively integrate visual information. Masking images during inference leads to minimal performance degradation, suggesting models heavily rely on text.
  • Task Formulation Matters: The "multimodal radiology report summarization" task might be ill-defined if the inherent nature of the data (e.g., comprehensive FINDINGS) makes one modality less critical. This suggests a need to reassess task definitions when adding modalities.
  • Two-Stage Fine-Tuning Shows Promise: A sequential fine-tuning approach (projection layer first, then language model) can encourage models to better align visual features with textual concepts, leading to improved image utilization.
  • Experts Agree with Models: Medical experts' annotation study corroborates the model findings, indicating they can often write IMPRESSIONS without the image, especially when the FINDINGS are detailed.
  • Missing Context: Experts noted that "imaging request" information, which guides the radiologist's focus, is often missing from datasets and could be crucial for more accurate summarization.

Key Results

  • Text-Only Models Often Outperform: In many evaluations, text-only models like BART performed as well as or better than multimodal models, especially on metrics like ROUGE and factual correctness.
  • Minimal Impact of Image Masking: Masking the radiology image input for multimodal models resulted in minimal to no performance drop across various metrics and datasets, indicating poor visual utilization.
  • Two-Stage Fine-Tuning Improves Utilization: Models fine-tuned with the two-stage strategy showed a significant performance drop when images were masked, suggesting better integration of visual information.
  • Expert Agreement: Medical experts agreed that radiology images were unnecessary for writing the IMPRESSION section 94% of the time for MIMIC-CXR and 88% for CheXpert.
  • LLM Evaluation: Using GPT-4 as a judge, the text-only BART model generated fewer clinically significant errors compared to LVLMs.

Limitations and Future Work

  • Domain Specificity: The study is limited to chest X-ray images and radiology reports. Findings may not generalize to other imaging modalities (e.g., CT, MRI) or other medical domains.
  • Dataset Bias: Datasets are primarily from US institutions, which might have reporting styles and clinical workflows that differ from other regions.
  • Future Work: The authors suggest future research should focus on:
  • Reassessing multimodal task definitions.
  • Addressing modality imbalance in datasets.
  • Involving expert annotation to better reflect real-world clinical decision-making.
  • Identifying specific cases where additional input beyond FINDINGS is truly necessary.
  • Investigating the impact of missing clinical context like "imaging requests."

Summary updated

BibTeX
@inproceedings{sim2025more,
  author = {Mong Yuan Sim and Wei Emma Zhang and Xiang Dai and Biaoyan Fang and Sarbin Ranjitkar and Arjun Burlakoti and Jamie Taylor and Haojie Zhuang},
  title = {{The More, The Better? A Critical Study of Multimodal Context in Radiology Report Summarization}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure