Introduction
Automating radiology report generation represents a significant opportunity to reduce physician burnout and improve diagnostic efficiency. However, existing Large Language Model (LLM) based systems face a critical hurdle: the modality gap. While LLMs are adept at generating fluent text, they often prioritize linguistic patterns over clinical reality when processing visual data. This leads to "hallucinations" where a model might produce a perfectly phrased sentence that contradicts the actual medical findings in an X-ray image.
To address this, the OTDRG (Optimal Transport-Driven Radiology Report Generation) framework introduces a mathematically rigorous method to align visual features with standardized medical concepts, ensuring that generated reports are both linguistically fluent and clinically accurate.
The OTDRG Framework
The core innovation of OTDRG lies in its ability to "textualize" visual information before it reaches the LLM. The framework bridges the vision-language gap through a multi-stage pipeline designed to enforce clinical guardrails.
Methodology Overview
- Disease Label Extraction (DLE): To provide a structured foundation, a frozen LLM extracts 14 standardized disease labels (e.g., Cardiomegaly, Edema) from existing reports. These labels act as anchors for the visual features.
- Feature Integration by Optimal Transport (FIOT): This is the system's centerpiece. By treating the alignment of image patches and disease labels as an Optimal Transport (OT) problem, the model uses the Sinkhorn-Knopp algorithm to find the most efficient mapping between visual regions and medical conditions. This ensures the LLM receives features that are already aligned with clinical concepts.
- Image-Report Contrastive Pre-training (IRCP): The visual encoder (Swin Transformer) undergoes pre-training to maximize similarity between images and their corresponding reports, sharpening the model's ability to detect relevant medical features.
- Efficient Fine-Tuning: The framework utilizes Low-Rank Adaptation (LoRA) to fine-tune the LLM, allowing it to incorporate medical domain knowledge without the prohibitive cost of retraining billions of parameters.
Key Insights and Results
OTDRG moves beyond standard NLP metrics like BLEU, which can be misleading in medical contexts (a report can be fluent but diagnostically wrong). Instead, the authors emphasize Clinical Efficacy (CE).
Performance Benchmarks
The model was evaluated on the MIMIC-CXR and IU X-Ray datasets, achieving state-of-the-art (SOTA) results:
- Clinical Accuracy: On MIMIC-CXR, OTDRG achieved a Clinical Efficacy F1 score of 0.436, significantly outperforming previous baselines.
- Linguistic Quality: The model maintained high fluency with a BLEU-1 score of 0.411.
- Ablation Studies: Removing the FIOT module led to a sharp decline in clinical precision, confirming that Optimal Transport is the primary driver for diagnostic accuracy.
Conclusion and Future Work
By mathematically aligning visual patches with clinical labels, OTDRG provides a robust solution to the problem of LLM hallucinations in medical imaging. While currently constrained to 14 primary disease groups, future iterations could expand this vocabulary to include rarer conditions and more granular anatomical descriptions. The framework demonstrates that rigorous feature alignment is essential for deploying generative AI in high-stakes clinical environments.
Summary updated