Bridging Vision and Language: Optimal Transport-Driven Radiology Report Generation via LLMs

Haifeng Zhao, Yufei Zhang, Leilei Ma, Shuo Xu, Dengdi SunarXiv 2025 · DOI · arXiv

Introduction

Automating radiology report generation represents a significant opportunity to reduce physician burnout and improve diagnostic efficiency. However, existing Large Language Model (LLM) based systems face a critical hurdle: the modality gap. While LLMs are adept at generating fluent text, they often prioritize linguistic patterns over clinical reality when processing visual data. This leads to "hallucinations" where a model might produce a perfectly phrased sentence that contradicts the actual medical findings in an X-ray image.

Comparison between raw LLM generation (hallucination) and the proposed aligned OTDRG framework.

To address this, the OTDRG (Optimal Transport-Driven Radiology Report Generation) framework introduces a mathematically rigorous method to align visual features with standardized medical concepts, ensuring that generated reports are both linguistically fluent and clinically accurate.

The OTDRG Framework

The core innovation of OTDRG lies in its ability to "textualize" visual information before it reaches the LLM. The framework bridges the vision-language gap through a multi-stage pipeline designed to enforce clinical guardrails.

Detailed architecture of the OTDRG framework, highlighting the FIOT module and contrastive pre-training stages.

Methodology Overview

  1. Disease Label Extraction (DLE): To provide a structured foundation, a frozen LLM extracts 14 standardized disease labels (e.g., Cardiomegaly, Edema) from existing reports. These labels act as anchors for the visual features.
  2. Feature Integration by Optimal Transport (FIOT): This is the system's centerpiece. By treating the alignment of image patches and disease labels as an Optimal Transport (OT) problem, the model uses the Sinkhorn-Knopp algorithm to find the most efficient mapping between visual regions and medical conditions. This ensures the LLM receives features that are already aligned with clinical concepts.
  3. Image-Report Contrastive Pre-training (IRCP): The visual encoder (Swin Transformer) undergoes pre-training to maximize similarity between images and their corresponding reports, sharpening the model's ability to detect relevant medical features.
  4. Efficient Fine-Tuning: The framework utilizes Low-Rank Adaptation (LoRA) to fine-tune the LLM, allowing it to incorporate medical domain knowledge without the prohibitive cost of retraining billions of parameters.

Key Insights and Results

OTDRG moves beyond standard NLP metrics like BLEU, which can be misleading in medical contexts (a report can be fluent but diagnostically wrong). Instead, the authors emphasize Clinical Efficacy (CE).

Performance Benchmarks

The model was evaluated on the MIMIC-CXR and IU X-Ray datasets, achieving state-of-the-art (SOTA) results:

  • Clinical Accuracy: On MIMIC-CXR, OTDRG achieved a Clinical Efficacy F1 score of 0.436, significantly outperforming previous baselines.
  • Linguistic Quality: The model maintained high fluency with a BLEU-1 score of 0.411.
  • Ablation Studies: Removing the FIOT module led to a sharp decline in clinical precision, confirming that Optimal Transport is the primary driver for diagnostic accuracy.

Conclusion and Future Work

By mathematically aligning visual patches with clinical labels, OTDRG provides a robust solution to the problem of LLM hallucinations in medical imaging. While currently constrained to 14 primary disease groups, future iterations could expand this vocabulary to include rarer conditions and more granular anatomical descriptions. The framework demonstrates that rigorous feature alignment is essential for deploying generative AI in high-stakes clinical environments.

Summary updated

Abstract

Radiology report generation represents a significant application within medical AI, and has achieved impressive results. Concurrently, large language models (LLMs) have demonstrated remarkable performance across various domains. However, empirical validation indicates that general LLMs tend to focus more on linguistic fluency rather than clinical effectiveness, and lack the ability to effectively capture the relationship between X-ray images and their corresponding texts, thus resulting in poor clinical practicability. To address these challenges, we propose Optimal Transport-Driven Radiology Report Generation (OTDRG), a novel framework that leverages Optimal Transport (OT) to align image features with disease labels extracted from reports, effectively bridging the cross-modal gap. The core component of OTDRG is Alignment \& Fine-Tuning, where OT utilizes results from the encoding of label features and image visual features to minimize cross-modal distances, then integrating image and text features for LLMs fine-tuning. Additionally, we design a novel disease prediction module to predict disease labels contained in X-ray images during validation and testing. Evaluated on the MIMIC-CXR and IU X-Ray datasets, OTDRG achieves state-of-the-art performance in both natural language generation (NLG) and clinical efficacy (CE) metrics, delivering reports that are not only linguistically coherent but also clinically accurate.
BibTeX
@misc{zhao2025bridging,
  author = {Haifeng Zhao and Yufei Zhang and Leilei Ma and Shuo Xu and Dengdi Sun},
  title = {{Bridging Vision and Language: Optimal Transport-Driven Radiology Report Generation via LLMs}},
  year = {2025},
  eprint = {2507.03908},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2507.03908},
}
Figure