TL;DR
InstructX2X is a novel local editing model for counterfactual medical image generation that prevents unintended changes to patient demographics (like age and race) while providing inherently interpretable visual guidance maps. By combining a region-specific editing mechanism with a new expert-verified dataset, it achieves state-of-the-art performance in generating highly realistic, clinically accurate chest X-rays.
What Problem Does This Paper Address?
Counterfactual image generation answers "what-if" questions in medicine, such as: "How would this patient's chest X-ray change if they developed pulmonary edema?" This capability is crucial for evaluating AI model robustness, uncovering causal structures, and providing visual explanations to clinicians.
However, current state-of-the-art generative models suffer from two major flaws:
- Collateral Modifications: When instructed to add or remove a disease feature, models often inadvertently alter independent patient attributes like age or race (e.g., changing a patient's apparent race from Black to White just to add an edema feature).
- Lack of Interpretability: Most models operate as black boxes, relying on unreliable post-hoc explanation tools that fail to reflect the model's actual internal decision-making process.
Methods
InstructX2X builds upon the text-guided image editing framework InstructPix2Pix, introducing a novel Region-Specific Editing (RSE) mechanism during inference to restrict modifications strictly to relevant anatomical areas.
The RSE pipeline operates as follows:
- Relevance Map Generation: The model computes the normalized absolute difference between noise predictions made with and without the text instruction. This highlights the regions the model intends to modify.
- Anatomically-Aware Pseudo Masks: The authors leverage expert-annotated bounding boxes to create localized masks for specific pathologies.
- Guidance Map ($G$): The relevance map and pseudo mask are combined via element-wise multiplication. This creates an inherently interpretable visual explanation of the editing process.
- Binary Masking: A threshold is applied to the Guidance Map to form a binary editing mask. During the diffusion denoising iterations, any pixels outside this mask are kept completely identical to the original input image, entirely eliminating background distortion and demographic drift.
Datasets
The model was trained and evaluated using three primary datasets:
- MIMIC-EDIT-INSTRUCTION: A new dataset introduced in this paper. The authors repurposed the existing MIMIC-Diff-VQA dataset (which contains longitudinal chest X-rays and expert-verified QA pairs) using a rule-based algorithm to extract precise anatomical locations and severity levels. It contains 21,957 high-quality samples across three core operations: Add, Remove, and Change the level.
- MS-CXR: Used to provide the expert-annotated bounding boxes needed to construct the anatomically-aware pseudo masks.
- MIMIC-CXR: The broader base dataset containing chest radiographs and radiology reports.
Key Insights and Ideas
- Spurious Correlations in Generative AI: Standard diffusion models easily exploit shortcuts. If a training dataset contains demographic biases (e.g., a specific disease appearing more frequently in a certain age group), the model will alter the patient's apparent age to match that bias when generating the disease feature.
- Inherent vs. Post-Hoc Interpretability: Instead of generating an image and then trying to explain it using secondary tools (post-hoc), InstructX2X integrates the explanation directly into the generation loop via the Guidance Map. The explanation perfectly matches the edit because the edit is physically constrained by the explanation.
- Repurposing VQA for Image Editing: Rather than relying on noisy, unverified Large Language Model (LLM) outputs to generate image-editing instructions, extracting instructions from expert-validated medical Visual Question Answering (VQA) datasets preserves strict clinical validity.
Key Results
InstructX2X was evaluated against leading baselines including RoentGen, RadEdit, and BiomedJourney:
- Superior Identity Preservation: InstructX2X achieved a comprehensive Counterfactual Medical Image Generation (CMIG) score of 89.35, closely approaching the ground-truth real image baseline of 90.99. It maintained a Race preservation score of 97.65 and an Age preservation score of 82.84, whereas baseline models like RoentGen suffered catastrophic drops (53.15 for race and 18.60 for age).
- High Clinical Realism: The model achieved the lowest KL divergence (7.88), indicating that its generated pathology distribution closely mirrors real clinical data rather than over-optimizing or inflating features.
- Visual Fidelity: It delivered an impressive FID score of 2.64, significantly outperforming BiomedJourney (12.46) and RoentGen (35.19).
Limitations and Future Work
The primary limitation noted is the model's dependency on the eight predefined pathology categories available in the MS-CXR dataset for pseudo-mask generation. For medical findings falling outside these eight categories, the model defaults to a full-sized ($512 \times 512$) mask, which temporarily removes the localization benefit of the RSE module.
Future work will focus on expanding the pseudo-mask generation framework to encompass a wider array of rare conditions and exploring automated segmentation models to dynamically define regions of interest.
Summary updated