MAIRA-1: A specialised large multimodal model for radiology report generation

Stephanie L. Hyland, Shruthi Bannur, Kenza Bouzid, Daniel C. Castro, Mercy Ranjit, Anton Schwaighofer, Fernando Pérez-García, Valentina Salvatelli, Shaury Srivastav, Anja Thieme, Noel Codella, Matthew P. Lungren, Maria Teodora Wetscherek, Ozan Oktay, Javier Alvarez-Valle

TL;DR

MAIRA-1 is a specialized large multimodal model (LMM) designed to generate the "Findings" section of chest X-ray reports. By combining a domain-specific vision encoder (RAD-DINO) with a fine-tuned Vicuna-7B LLM and leveraging GPT-based data augmentation, it achieves state-of-the-art performance on both lexical and clinical metrics, significantly outperforming general-purpose models like LLaVA.

What Problem Does This Paper Address?

Generating radiology reports is significantly more complex than standard image captioning. A radiologist doesn't just describe what is there; they must also confirm the absence of critical findings (e.g., "no pneumothorax"), identify subtle variations in opacity against complex anatomical backgrounds, and synthesize the image with clinical context (the "Indication").

General-domain multimodal models typically fail at this task because they lack the fine-grained visual sensitivity required for medical imaging and the specialized vocabulary of radiology. Furthermore, existing evaluation metrics often fail to capture clinical accuracy, making it difficult to track real progress in the field.

Methods

The authors adopt the LLaVA-style architecture—aligning a vision encoder to a Large Language Model (LLM) via an adapter—but introduce several domain-specific optimizations:

  1. Specialized Vision Encoder: Instead of using a general-purpose CLIP encoder, they use RAD-DINO, a ViT-B model pretrained on 838,000 chest X-rays. Crucially, it operates at a higher resolution (518x518) compared to the standard 224px or 336px, which is vital for spotting small abnormalities.
  2. Deeper Adapter: They scaled the adapter from the standard two-layer MLP to a four-layer MLP to better handle the increased number of image tokens (1,369 tokens) generated by the high-resolution encoder.
  3. Clinical Context Integration: The model is prompted with both the image and the Indication section (the clinical reason for the scan), mimicking the information available to a human radiologist.
  4. GPT-Augmentation: To prevent the model from overfitting to the specific formulaic style of the training data, they used GPT-3.5 to paraphrase the training reports. This encourages the model to focus on clinical semantics rather than rote memorization of phrases.

Datasets

The study primarily utilizes the MIMIC-CXR dataset, a massive public repository of chest X-rays and reports.

  • Training/Validation: The authors processed DICOM images (rather than lower-quality JPEGs) to preserve detail and filtered for frontal views (AP/PA).
  • Data Augmentation: An additional 131,558 reports were generated via GPT-3.5 paraphrasing for the training set.
  • Evaluation: They used the standard MIMIC-CXR test split but carefully excluded specific benchmark subsets (like MS-CXR) from training to ensure a clean evaluation.
Table 1: Number of subjects, studies and DICOMs in each of our splits of the MIMIC-CXR dataset.

Key Insights and Ideas

  • Domain Specificity Trumps Scale: A smaller, domain-specific vision encoder (RAD-DINO) outperformed much larger general-purpose encoders. The authors found that even if a model was pretrained on biomedical data (like LLaVA-Med), it still lagged behind MAIRA-1's specialized pipeline.
  • The "Indication" is Essential: Including the clinical reason for the study significantly boosted performance. It provides the model with "cues" on what to look for and, importantly, what to report as absent.
  • Resolution is a Bottleneck: The jump to 518px resolution was a key driver in performance, likely because many radiographic findings are too subtle to survive the downsampling typical of general-purpose LMMs.
  • Paraphrasing as a Regularizer: Using LLMs to paraphrase medical text (as shown in Figure 2) helps the model learn the underlying clinical concepts rather than just mimicking the "noisy" and often inconsistent writing styles found in real-world hospital data.
Table 2: Example of the output of GPT-3.5 paraphrasing on the Indication and Findings sections. Paraphrasing preserves the clinical content while introducing small variations in phrasing.

Key Results

MAIRA-1 achieved state-of-the-art (SOTA) results across nearly all metrics considered.

  • Lexical Metrics: It set new benchmarks for ROUGE-L (28.9) and BLEU-1 (39.2).
  • Clinical Metrics: Most notably, it improved the RadCliQ score to 3.10. RadCliQ is a composite metric specifically designed to align with the number of errors a human radiologist would find in a report.
  • Ablation Success: As shown in Figure 4, each design choice—moving to RAD-DINO, increasing adapter size, and adding GPT-augmented data—provided additive gains to the model's clinical accuracy.
Table 4: Effect of model design choices on findings generation performance. We report median and 95% confidence intervals based on 500 bootstrap samples from the MIMIC-CXR test set. (a) Comparison of our approach to 3 baselines based on continually training LLaVA-style models (b) Additive gains from using RAD-DINO, increasing the adapter size, and adding GPT-augmented data. Bold indicates best performance across all experiments. ‘↓’ indicates that lower is better (RadCliQ only). CheXpert F1 metrics are computed based on CheXbert labeller ouputs.

Compared to other SOTA models (Table 5), MAIRA-1 remains highly competitive despite having a much smaller parameter count in its vision component than models like Med-PaLM M.

Table 5: Findings generation performance on the MIMIC-CXR test set, compared to closest state-of-the-art for each metric. We report median with 95% confidence intervals from 500 bootstrap samples of the test-set for MAIRA-1. Prior work uses differing test sets, limiting comparison. Our model is composed of a 86.6M parameter image encoder, 53M MLP adapter and 7B LLM. Our test size is 2461 samples. For the CheXpert F1 metrics, ‘+’ means the uncertain class is mapped to positive, otherwise it is mapped to negative. *The CheXbert vector score is an evaluation of the Miura et al. [2021] model performed by Yu et al. [2023]. †The RadCliQ number is an evaluation of Miura et al. [2021] model reported by Jeong et al. [2023], Yu et al. [2023], using RadCliQ-v0.

Limitations and Future Work

  • Single-Image Constraint: Radiologists rarely interpret a scan in a vacuum; they compare it to "priors" (previous scans). Because MAIRA-1 only sees one image, it sometimes "hallucinates" comparisons to non-existent prior studies.
  • Quantitative Grounding: The model occasionally generates specific measurements (e.g., "tube is 4cm above the carina") that are not grounded in actual physical pixel-to-cm mapping, but rather learned correlations.
  • Rare Findings: While the model is excellent at identifying common issues like pleural effusion, it still has lower recall for rarer or more clinically complex diagnoses like pneumonia.
  • Future Directions: The authors suggest that future versions should incorporate multiple images (priors and lateral views) and richer clinical histories to further reduce hallucinations and improve diagnostic accuracy.

Summary updated

Abstract

We present a radiology-specific multimodal model for the task for generating radiological reports from chest X-rays (CXRs). Our work builds on the idea that large language model(s) can be equipped with multimodal capabilities through alignment with pre-trained vision encoders. On natural images, this has been shown to allow multimodal models to gain image understanding and description capabilities. Our proposed model (MAIRA-1) leverages a CXR-specific image encoder in conjunction with a fine-tuned large language model based on Vicuna-7B, and text-based data augmentation, to produce reports with state-of-the-art quality. In particular, MAIRA-1 significantly improves on the radiologist-aligned RadCliQ metric and across all lexical metrics considered. Manual review of model outputs demonstrates promising fluency and accuracy of generated reports while uncovering failure modes not captured by existing evaluation practices. More information and resources can be found on the project website: https://aka.ms/maira.
BibTeX
@misc{hyland2023maira,
  author = {Stephanie L. Hyland and Shruthi Bannur and Kenza Bouzid and Daniel C. Castro and Mercy Ranjit and Anton Schwaighofer and Fernando Pérez-García and Valentina Salvatelli and Shaury Srivastav and Anja Thieme and Noel Codella and Matthew P. Lungren and Maria Teodora Wetscherek and Ozan Oktay and Javier Alvarez-Valle},
  title = {{MAIRA-1: A specialised large multimodal model for radiology report generation}},
  year = {2023},
  eprint = {2311.13668},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2311.13668},
}
Figure