MediVLM: A Vision Language Model for Radiology Report Generation from Medical Images

Debanjan Goswami, Ronast Subedi, Shayok ChakrabortyEMNLP 2025 · OpenReview

TL;DR

MediVLM is a novel vision-language model that generates radiology reports from medical images, addressing key limitations of prior work by focusing on salient image regions and enabling unsupervised training. It also introduces a useful severity score to help prioritize patient care.

What Problem Does This Paper Address?

Generating accurate and informative radiology reports from medical images is crucial for patient diagnosis and care. However, existing automated methods often struggle with:

  • Incompleteness and Inconsistency: Reports might miss important findings or contain factual errors.
  • Lack of Focus: Models tend to process the entire image, failing to pinpoint specific anatomical regions where abnormalities lie.
  • Data Dependency: Many methods require extensive, fine-grained annotations (like bounding boxes), which are costly and time-consuming to obtain, especially given the shortage of trained radiologists.
  • Need for Paired Data: Training typically requires images paired with corresponding reports, a bottleneck when such data is scarce.

This paper aims to overcome these challenges by developing a more robust, flexible, and data-efficient system for radiology report generation.

Methods

MediVLM employs a multi-stage architecture:

  1. Salient Region Extraction: A pre-trained Faster R-CNN object detector identifies key anatomical regions of interest within the medical image. This allows the model to focus its attention on potentially abnormal areas.
  2. Image and Text Encoding:
    • Image Encoder: A frozen CLIP-ViT model extracts spatial features from the identified image patches. Position embeddings are also incorporated to retain spatial information.
  3. Text Encoder: A frozen ClinicalBERT model, specialized for clinical text, is used to embed input radiology reports into latent representations.
  4. Alignment and Fusion:
    • Contrastive Learning: A contrastive loss function aligns the visual and textual embeddings, ensuring that corresponding image-text pairs are mapped to similar representations.
  5. Cross-Attention: A cross-attention layer fuses these aligned representations into a single latent representation.
  6. Report Generation: A fine-tuned GPT-2 transformer decoder generates the final radiology report token by token.
  7. Severity Score Computation: A novel feature, MediVLM computes a severity score for each generated report using TF-IDF scores of predefined severity-related terms, helping to prioritize critical cases.

A key innovation is the unsupervised training capability. MediVLM can learn to generate reports even when only images are available, by using a pseudo-labeling strategy to create synthetic reports for training.

Datasets

The authors evaluated MediVLM on three benchmark datasets:

  • IU X-Ray: A public radiology dataset from Indiana University.
  • CASIA-CXR: A dataset of chest radiographs with reports written in French (specifically the Pneumonia category was used).
  • MIMIC-CXR: A large, publicly available dataset of chest radiographs with free-text radiology reports.

Key Insights and Ideas

  • Focusing on Salient Regions is Crucial: Using an object detector to extract specific regions of interest significantly improves report generation performance compared to processing the entire image. This highlights the importance of targeted analysis in medical imaging.
  • ClinicalBERT's Advantage: Employing ClinicalBERT as the text encoder, rather than a general-purpose one like CLIP's text encoder, leads to better text embeddings and improved overall performance, underscoring the value of domain-specific models.
  • Unsupervised Training is a Game-Changer: The ability to train effectively without paired image-report data is a significant advancement, addressing a major bottleneck in medical AI development and making the technology more accessible.
  • Severity Score for Prioritization: The integrated severity score is a practical addition that can directly aid clinical workflows by helping to identify and prioritize patients requiring immediate attention.

Key Results

MediVLM demonstrated strong performance across various metrics:

  • Supervised Learning: MediVLM comprehensively outperformed competing baselines on the IU X-Ray dataset across multiple natural language generation (BLEU, METEOR, ROUGE-L) and clinical relevance metrics (BERTScore, RadGraph-F1, RaTEScore). It also achieved competitive results on CASIA-CXR and MIMIC-CXR.
  • Unsupervised Learning: In the challenging unsupervised setting, MediVLM outperformed the RL baseline on CASIA-CXR and showed competitive results on IU X-Ray, validating the efficacy of its pseudo-labeling pipeline.
  • Ablation Studies: The studies confirmed the benefits of using salient image patches and the ClinicalBERT text encoder.

Limitations and Future Work

The authors acknowledge the following limitations:

  • Lack of Clinical Validation: The generated reports and severity scores have not yet been validated by medical professionals.
  • Domain Specificity: The current framework is primarily tested on Chest X-ray images.

Future work includes:

  • Qualitative User Studies: Engaging domain experts to validate the usefulness of MediVLM for clinical applications.
  • Broader Applicability: Extending the framework to other types of medical images beyond Chest X-rays.

Summary updated

Abstract

Generating radiology reports from medical images has garnered sufficient attention in the research community. While existing methods have demonstrated promise, they often tend to generate reports that are factually incomplete and inconsistent, fail to focus on informative regions within an image, and impose strong annotation assumptions, such as bounding box annotations, image level annotations (which can be challenging to obtain) for model training. In this paper, we propose MediVLM, a vision language model (VLM) for radiology report generation from medical images. The proposed model consists of a pre-trained object detector to extract the salient anatomical regions from the images, an image encoder, a text encoder, a module to align the visual and text representations, a cross attention layer to fuse the two representations and finally, a transformer based decoder to generate the final report. MediVLM can generate radiology reports even when no reports are available for training; this is an extremely useful feature, as curating such reports is a labor-intensive task. Further, it computes a severity score (depicting the seriousness of a patient’s medical condition) from the generated radiology reports, which can be used to prioritize patients who need immediate medical attention. Our extensive empirical analyses on three benchmark datasets corroborate the promise and potential of our method against competing baselines. Our code is opensourced in our project webpage at: https: //sites.google.com/view/medivlm/home
BibTeX
@inproceedings{goswami2025medivlm,
  author = {Debanjan Goswami and Ronast Subedi and Shayok Chakraborty},
  title = {{MediVLM: A Vision Language Model for Radiology Report Generation from Medical Images}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  url = {https://openreview.net/forum?id=6jOGu86C2j},
}
Figure