An Interpretable Local Editing Model for Counterfactual Medical Image Generation

Hyungi Min, Taeseung You, Hangyeul Lee, Yeongjae Cho, Sungzoon ChoarXiv 2026 · arXiv

TL;DR

InstructX2X is a novel local editing model for counterfactual medical image generation that prevents unintended changes to patient demographics (like age and race) while providing inherently interpretable visual guidance maps. By combining a region-specific editing mechanism with a new expert-verified dataset, it achieves state-of-the-art performance in generating highly realistic, clinically accurate chest X-rays.

What Problem Does This Paper Address?

Counterfactual image generation answers "what-if" questions in medicine, such as: "How would this patient's chest X-ray change if they developed pulmonary edema?" This capability is crucial for evaluating AI model robustness, uncovering causal structures, and providing visual explanations to clinicians.

However, current state-of-the-art generative models suffer from two major flaws:

  1. Collateral Modifications: When instructed to add or remove a disease feature, models often inadvertently alter independent patient attributes like age or race (e.g., changing a patient's apparent race from Black to White just to add an edema feature).
  2. Lack of Interpretability: Most models operate as black boxes, relying on unreliable post-hoc explanation tools that fail to reflect the model's actual internal decision-making process.
Comparison of counterfactual medical image generation results between existing methods and InstructX2X

Methods

InstructX2X builds upon the text-guided image editing framework InstructPix2Pix, introducing a novel Region-Specific Editing (RSE) mechanism during inference to restrict modifications strictly to relevant anatomical areas.

The RSE pipeline operates as follows:

  • Relevance Map Generation: The model computes the normalized absolute difference between noise predictions made with and without the text instruction. This highlights the regions the model intends to modify.
  • Anatomically-Aware Pseudo Masks: The authors leverage expert-annotated bounding boxes to create localized masks for specific pathologies.
  • Guidance Map ($G$): The relevance map and pseudo mask are combined via element-wise multiplication. This creates an inherently interpretable visual explanation of the editing process.
  • Binary Masking: A threshold is applied to the Guidance Map to form a binary editing mask. During the diffusion denoising iterations, any pixels outside this mask are kept completely identical to the original input image, entirely eliminating background distortion and demographic drift.
Figure 2: Overview of the InstructX2X training framework. The pipeline is adapted from the InstructPix2Pix architecture [17], modified to accept longitudinal chest X-ray pairs. The top panel illustrates the dataset construction process converting descriptions pairs into the MIMIC-EDIT-INSTRUCTION data. The bottom panel shows the training pipeline where the model learns to transform Ipast to Icur using the constructed instruc- tions.

Datasets

The model was trained and evaluated using three primary datasets:

  • MIMIC-EDIT-INSTRUCTION: A new dataset introduced in this paper. The authors repurposed the existing MIMIC-Diff-VQA dataset (which contains longitudinal chest X-rays and expert-verified QA pairs) using a rule-based algorithm to extract precise anatomical locations and severity levels. It contains 21,957 high-quality samples across three core operations: Add, Remove, and Change the level.
  • MS-CXR: Used to provide the expert-annotated bounding boxes needed to construct the anatomically-aware pseudo masks.
  • MIMIC-CXR: The broader base dataset containing chest radiographs and radiology reports.

Key Insights and Ideas

  • Spurious Correlations in Generative AI: Standard diffusion models easily exploit shortcuts. If a training dataset contains demographic biases (e.g., a specific disease appearing more frequently in a certain age group), the model will alter the patient's apparent age to match that bias when generating the disease feature.
  • Inherent vs. Post-Hoc Interpretability: Instead of generating an image and then trying to explain it using secondary tools (post-hoc), InstructX2X integrates the explanation directly into the generation loop via the Guidance Map. The explanation perfectly matches the edit because the edit is physically constrained by the explanation.
  • Repurposing VQA for Image Editing: Rather than relying on noisy, unverified Large Language Model (LLM) outputs to generate image-editing instructions, extracting instructions from expert-validated medical Visual Question Answering (VQA) datasets preserves strict clinical validity.

Key Results

InstructX2X was evaluated against leading baselines including RoentGen, RadEdit, and BiomedJourney:

  • Superior Identity Preservation: InstructX2X achieved a comprehensive Counterfactual Medical Image Generation (CMIG) score of 89.35, closely approaching the ground-truth real image baseline of 90.99. It maintained a Race preservation score of 97.65 and an Age preservation score of 82.84, whereas baseline models like RoentGen suffered catastrophic drops (53.15 for race and 18.60 for age).
  • High Clinical Realism: The model achieved the lowest KL divergence (7.88), indicating that its generated pathology distribution closely mirrors real clinical data rather than over-optimizing or inflating features.
  • Visual Fidelity: It delivered an impressive FID score of 2.64, significantly outperforming BiomedJourney (12.46) and RoentGen (35.19).

Limitations and Future Work

The primary limitation noted is the model's dependency on the eight predefined pathology categories available in the MS-CXR dataset for pseudo-mask generation. For medical findings falling outside these eight categories, the model defaults to a full-sized ($512 \times 512$) mask, which temporarily removes the localization benefit of the RSE module.

Future work will focus on expanding the pseudo-mask generation framework to encompass a wider array of rare conditions and exploring automated segmentation models to dynamically define regions of interest.

Summary updated

Abstract

Counterfactual medical image generation have emerged as a critical tool for enhancing AI-driven systems in medical domain by answering "what-if" questions. However, existing approaches face two fundamental limitations: First, they fail to prevent unintended modifications, resulting collateral changes in demographic attributes when only disease features should be affected. Second, they lack interpretability in their editing process, which significantly limits their utility in real-world medical applications. To address these limitations, we present InstructX2X, a novel interpretable local editing model for counterfactual medical image generation featuring Region-Specific Editing. This approach restricts modifications to specific regions, effectively preventing unintended changes while simultaneously providing a Guidance Map that offers inherently interpretable visual explanations of the editing process. Additionally, we introduce MIMIC-EDIT-INSTRUCTION, a dataset for counterfactual medical image generation derived from expert-verified medical VQA pairs. Through extensive experiments, InstructX2X achieve state-of-the-art performance across all major evaluation metrics. Our model successfully generates high-quality counterfactual chest X-ray images along with interpretable explanations.
BibTeX
@misc{min2026interpretable,
  author = {Hyungi Min and Taeseung You and Hangyeul Lee and Yeongjae Cho and Sungzoon Cho},
  title = {{An Interpretable Local Editing Model for Counterfactual Medical Image Generation}},
  year = {2026},
  eprint = {2603.00423},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.00423},
}

Loading related content...

Figure