Segmentor-Guided Counterfactual Fine-Tuning for Locally Coherent and Targeted Image Synthesis

Tian Xia, Matthew Sinclair, Andreas Schuh, Fabio De Sousa Ribeiro, Raghav Mehta, Rajat Rasal, Esther Puyol-Antón, Samuel Gerber, Kersten Petersen, Michiel Schaap, Ben Glocker

Introduction

In medical AI, generating "what-if" (counterfactual) images—such as visualizing a chest X-ray with a smaller heart or an artery without plaque—is essential for training robust models, explaining AI decisions, and predicting disease progression. Current state-of-the-art methods utilize Deep Structural Causal Models (DSCMs) combined with Counterfactual Fine-Tuning (CFT). Traditionally, CFT relies on scalar regressors to guide the generative model toward a target attribute (e.g., a specific lung area). However, these regressors often learn spurious correlations, leading to global, unrealistic image alterations—like darkening the entire image to represent "smaller lungs"—rather than precise anatomical changes.

Methodology: Seg-CFT

To address these limitations, we introduce Segmentor-guided Counterfactual Fine-Tuning (Seg-CFT). Unlike standard CFT which uses a scalar regressor, Seg-CFT employs a pre-trained, weight-frozen segmentation model to provide spatial supervision during the fine-tuning phase.

The process involves four key steps:

  1. The generative model (a Hierarchical Variational Auto-Encoder) produces a counterfactual image based on an intervention.
  2. The frozen segmentor generates a pixel-level mask of the target structure.
  3. The area is calculated directly from this mask.
  4. The model is penalized if the calculated area deviates from the target value.

By forcing the model to generate images that a segmentor can accurately mask, the system learns the spatial boundaries and context of the anatomy. At inference time, the segmentor is no longer needed; users can simply adjust scalar values to generate counterfactuals.

Datasets and Evaluation

The method was evaluated on two medical datasets:

  • PadChest: Approximately 85,000 chest radiographs, focusing on the heart and lungs.
  • CCTA (Coronary Computed Tomography Angiography): Over 18,000 images used to model coronary artery disease, specifically arterial plaque and lumen area.

Results

Seg-CFT demonstrates significant improvements in both quantitative accuracy and qualitative localization.

Quantitative Accuracy

On the PadChest dataset, Seg-CFT achieved the lowest Mean Absolute Percentage Error (MAPE) across all interventions. For example, in left lung area interventions, Seg-CFT recorded a 10.0% error, outperforming the standard Reg-CFT (13.4%) and the baseline without fine-tuning (16.1%).

Qualitative Localization

Visual analysis using difference maps ($x - \tilde{x}$) confirms that Seg-CFT's modifications are tightly confined to the target anatomy.

In the CCTA study, Seg-CFT successfully simulated the addition or removal of arterial plaque without distorting vessel walls, a prerequisite for clinical utility.

Discussion and Future Work

While Seg-CFT offers a robust solution for 2D targeted image synthesis, several avenues for future research remain. Extending the framework to 3D volumetric data (CT/MRI) is a priority. Additionally, current models assume independence between anatomical structures; future iterations could incorporate more complex causal graphs to reflect biological reality. Finally, the guidance mechanism could be expanded beyond "area" to control shape, texture, or lesion placement.

Summary updated

BibTeX
@inproceedings{xia2025segmentor,
  author = {Tian Xia and Matthew Sinclair and Andreas Schuh and Fabio De Sousa Ribeiro and Raghav Mehta and Rajat Rasal and Esther Puyol-Antón and Samuel Gerber and Kersten Petersen and Michiel Schaap and Ben Glocker},
  title = {{Segmentor-Guided Counterfactual Fine-Tuning for Locally Coherent and Targeted Image Synthesis}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2025},
  doi = {10.1007/978-3-032-04937-7\_50},
}
Figure