Introduction
In medical AI, generating "what-if" (counterfactual) images—such as visualizing a chest X-ray with a smaller heart or an artery without plaque—is essential for training robust models, explaining AI decisions, and predicting disease progression. Current state-of-the-art methods utilize Deep Structural Causal Models (DSCMs) combined with Counterfactual Fine-Tuning (CFT). Traditionally, CFT relies on scalar regressors to guide the generative model toward a target attribute (e.g., a specific lung area). However, these regressors often learn spurious correlations, leading to global, unrealistic image alterations—like darkening the entire image to represent "smaller lungs"—rather than precise anatomical changes.
Methodology: Seg-CFT
To address these limitations, we introduce Segmentor-guided Counterfactual Fine-Tuning (Seg-CFT). Unlike standard CFT which uses a scalar regressor, Seg-CFT employs a pre-trained, weight-frozen segmentation model to provide spatial supervision during the fine-tuning phase.
The process involves four key steps:
- The generative model (a Hierarchical Variational Auto-Encoder) produces a counterfactual image based on an intervention.
- The frozen segmentor generates a pixel-level mask of the target structure.
- The area is calculated directly from this mask.
- The model is penalized if the calculated area deviates from the target value.
By forcing the model to generate images that a segmentor can accurately mask, the system learns the spatial boundaries and context of the anatomy. At inference time, the segmentor is no longer needed; users can simply adjust scalar values to generate counterfactuals.
Datasets and Evaluation
The method was evaluated on two medical datasets:
- PadChest: Approximately 85,000 chest radiographs, focusing on the heart and lungs.
- CCTA (Coronary Computed Tomography Angiography): Over 18,000 images used to model coronary artery disease, specifically arterial plaque and lumen area.
Results
Seg-CFT demonstrates significant improvements in both quantitative accuracy and qualitative localization.
Quantitative Accuracy
On the PadChest dataset, Seg-CFT achieved the lowest Mean Absolute Percentage Error (MAPE) across all interventions. For example, in left lung area interventions, Seg-CFT recorded a 10.0% error, outperforming the standard Reg-CFT (13.4%) and the baseline without fine-tuning (16.1%).
Qualitative Localization
Visual analysis using difference maps ($x - \tilde{x}$) confirms that Seg-CFT's modifications are tightly confined to the target anatomy.
In the CCTA study, Seg-CFT successfully simulated the addition or removal of arterial plaque without distorting vessel walls, a prerequisite for clinical utility.
Discussion and Future Work
While Seg-CFT offers a robust solution for 2D targeted image synthesis, several avenues for future research remain. Extending the framework to 3D volumetric data (CT/MRI) is a priority. Additionally, current models assume independence between anatomical structures; future iterations could incorporate more complex causal graphs to reflect biological reality. Finally, the guidance mechanism could be expanded beyond "area" to control shape, texture, or lesion placement.
Summary updated