Abstract
Medical report generation (MRG) is essential for computer-aided diagnosis and medication guidance, which can relieve the heavy burden of radiologists by automatically generating the corresponding medical reports according to the given radiology image. However, due to the spurious correlations within image-text data induced by visual and linguistic biases, it is challenging to generate accurate reports reliably describing lesion areas. Moreover, the cross-modal confounders are usually unobservable and challenging to be eliminated in an explicit way. In this paper, we aim to mitigate the cross-modal data bias for MRG from a new perspective, i.e., cross-modal causal intervention, and propose a novel Visual-Linguistic Causal Intervention (VLCI) framework for MRG, which consists of a visual deconfounding module (VDM) and a linguistic deconfounding module (LDM), to implicitly mitigate the visual-linguistic confounders by causal front-door intervention. Specifically, due to the absence of a generalized semantic extractor, the VDM explores and disentangles the visual confounders from the patch-based local and global features without expensive fine-grained annotations. Simultaneously, due to the lack of knowledge encompassing the entire medicine, the LDM eliminates the linguistic confounders caused by salient visual features and high-frequency context without constructing a terminology database. Extensive experiments on IU-Xray and MIMIC-CXR datasets show that our VLCI significantly outperforms the state-of-the-art MRG methods.
Introduction
Automating radiology report generation (RRG) is a "holy grail" for clinical AI—it promises to reduce radiologist burnout and catch subtle abnormalities that human eyes might miss. However, current models often "cheat." Instead of truly analyzing an image for disease markers, they rely on spurious correlations: statistical shortcuts found in training data. For instance, if a dataset frequently pairs a "Normal Heart" with "No Effusion," a model might learn to ignore actual signs of effusion simply because the heart looks healthy. This research introduces a framework called CMCRL (Cross-Modal Causal Representation Learning) to break these biased shortcuts using the power of causal inference.
The Challenge of Visual-Linguistic Bias
The core challenge in RRG is visual-linguistic bias. Medical datasets are often imbalanced; certain "normal" findings appear far more frequently than specific pathologies. Standard deep learning models are "confounded" by these patterns, leading them to generate reports based on the most likely co-occurrences rather than the actual evidence in the image. Furthermore, radiological images are often low-resolution or noisy, making it difficult for standard vision-language models to capture the fine-grained details necessary for a correct diagnosis.
The CMCRL Framework
The authors tackle these issues with a two-stage approach: RadCARE for pre-training and VLCI for fine-tuning with causal intervention.
RadCARE: Degradation-Aware Pre-training
To handle noisy, low-resolution data, the authors introduced RadCARE, which utilizes a "degradation-aware" masked image restoration strategy. The model learns to reconstruct high-resolution patches from low-resolution inputs, making it more sensitive to subtle lesions. This is paired with a multiway transformer architecture that can handle various data scenarios, such as missing images or missing text, during training.
VLCI: Visual-Linguistic Causal Intervention
This is the heart of the framework. Instead of simple back-propagation, the model uses Causal Front-Door Intervention through two internal mediators:
- Visual Deconfounding Module (VDM): Disentangles local details (like lung texture) from global structures (like heart contour).
- Linguistic Deconfounding Module (LDM): Adjusts for word frequency biases in the embedding space.
By forcing the model's prediction to pass through these mediators, the framework effectively "blocks" the influence of unobservable confounders that create spurious correlations.
Experimental Results
The CMCRL framework was evaluated on the IU-Xray and MIMIC-CXR datasets, achieving state-of-the-art (SOTA) results across multiple metrics. On the massive MIMIC-CXR dataset, it reached a BLEU-4 score of 11.9, outperforming previous heavyweight models.
More importantly, the model showed significant gains in Clinical Efficacy (CE), which measures the diagnostic accuracy of the generated text. CMCRL achieved a Precision of 48.9% and an F1-Score of 40.1%, a substantial improvement over the baseline R2Gen model (33.3% and 27.6%, respectively). Qualitative case studies confirm that the model correctly identifies pathologies like "Pleural Effusion" even when confounding factors (like a "Normal Heart") are present.
Conclusion
By shifting the focus from statistical correlation to causal reasoning, the CMCRL framework provides a more reliable path for automated radiology reporting. While the model still faces challenges with highly specialized or latent concepts, future work integrating Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG) promises to further enhance the clinical utility of these systems.
Summary updated