A recent family of methods finds lesions without anyone drawing them. A generator is trained only on scans tagged "diseased" or "healthy", and it learns to produce a "healed" version of each diseased scan. Subtracting the healed image from the original leaves the lesion. COIN, PathoGAN and diffusion-based variants all work this way. They are trained and tested inside one hospital. When the same generator meets a scan from a new scanner, nothing tells it that a change in texture is not disease, so it can "heal" the scanner's look instead of the tumor. The resulting mask then fails exactly where deployment happens. A related problem appears with rare sites and rare diseases: the few diseased scans from an underrepresented hospital are the ones the generator understands worst.
We propose to condition one generator on two labels that hospitals already record for free: disease status at the image level, and the acquisition site. We then require the two counterfactual edits to commute. If a scan is re-rendered as if it came from another hospital, its lesion mask should not move, because a scanner change alters appearance and not anatomy. This rule gives three things at once. It provides a test-time check that flags unreliable masks. It provides a training signal that keeps disease edits separate from scanner edits. It also provides a way to move diseased scans from common sites into rare sites with their masks attached, which yields site- and class-balanced segmentation training data without a single expert contour.
Core Claim
The expected gain from enforcing site commutation should be largest on two kinds of test data: centers never seen in training, and the rarest center×disease cells. On in-distribution test data it should help little. If commutation helps everywhere equally, it is acting as generic regularization rather than separating disease from scanner, and the claim fails. What would be new is using site counterfactuals as a consistency constraint and a data source for counterfactual segmentation from image-level labels. Prior counterfactual segmenters use a single domain. Prior site-aware generators need dense masks.
Relation to Prior Work
Three papers supply the core mechanism, and all three assume a single homogeneous domain. COIN trains a perturbation generator that flips a frozen classifier from abnormal to normal and reads the difference map as a mask. PathoGAN does the same with a CycleGAN that blends an inpainting into the scan through a learned label map, reporting about 72.9% Dice on BraTS against 86.3% for a supervised model. Conditional Diffusion Models for Weakly Supervised Medical Image Segmentation perturbs the class-condition embedding of a diffusion model, a generator that learns to turn noise into images step by step. It reads the change in predicted noise as a mask, reaching 0.563 Dice on BraTS in 2.78 seconds per image. We reuse this last method's fast finite-difference trick as our map extractor. Our contribution is to add a site axis to its condition and constrain how the two axes interact. Hide-and-Seek Attribution motivates the small-lesion part of the evaluation. Its healthy edits reach 0.87 instance Dice on blastic metastases but 0.78 on subtle lytic ones, which shows that low-contrast lesions are where counterfactual masks are most easily confused with background variation. That is the same confusion a scanner shift causes.
Two papers constrain the design. Red-GAN conditions a generator on site metadata and improves Dice on the rarest BraTS center by about 5% through targeted synthesis. However, its generator is driven by dense ground-truth masks, so it cannot ease the annotation bottleneck. We keep its insight that rare sites benefit from targeted synthesis and drop its need for masks. AMOS documents drops above 20% on unseen MRI scanners even with standard intensity preprocessing. This is the evidence that the problem is real and that simple normalization is not enough.
Two papers motivate the commutation idea from other angles. Unsupervised Structural-Counterfactual Generation under Domain Shift formalizes domain shift as separating domain-invariant identity factors from domain-specific noise, but validates only on low-dimensional synthetic equations. We test a weak, image-scale version of its assumption: anatomy is invariant, appearance is site-specific. SpurCon shows that pairs of images that differ only in a nuisance factor can fix shortcut learning, raising worst-group accuracy on ISIC 2020 from 0% to 65.0%. It builds those pairs from repeated scans of the same patient. Site counterfactuals generate such pairs where no patient appears at two hospitals.
Research Program
We train one conditional diffusion model on 2D slices, conditioned on a disease flag $d \in \{0,1\}$ and a site code $s$. Training uses only slice-level disease tags and site metadata. Write $m_s(x)$ for the lesion map extracted from image $x$ by perturbing the disease condition while holding site $s$ fixed. Write $T_{s \to s'}(x)$ for the site counterfactual: invert $x$ at noise level $Q$ under condition $(d,s)$ and regenerate under $(d,s')$. The commutation discrepancy for an image is
$$D(x) = \frac{1}{|S|}\sum_{s' \in S} \big\| m_s(x) - m_{s'}\!\big(T_{s \to s'}(x)\big) \big\|_1 ,$$
where $S$ is the set of training sites. A small $D(x)$ means the lesion map survives re-rendering at every hospital.
The program has three stages, each a separable claim.
The first stage uses commutation only at inference and requires no extra training. The final mask is the average of $m_{s'}(T_{s \to s'}(x))$ over sites, and $D(x)$ serves as a per-image confidence score. For an image from an unseen center, the nearest training site is chosen with a site classifier.
The second stage adds $D$ as a fine-tuning loss on the diffusion model, through short truncated denoising chains to keep the cost bounded. Two cross-checks enter the same loss. A frozen site classifier must not change its prediction under a disease edit, and a frozen disease classifier must not change its prediction under a site edit.
The third stage builds balanced training data. Diseased slices from common centers are site-translated into rare centers. Each translated slice carries the pseudo-mask of its source, and only pairs with low $D$ are kept. A standard segmentation U-Net is then trained on real and translated pseudo-labeled slices, sampled so that every center×disease cell is equally represented.
The program also addresses the synthesis dilemma raised by Red-GAN. We never ask the generator to invent rare lesions. Lesion shape always comes from a real diseased scan. Only appearance is borrowed from the rare site, and appearance can be learned from that site's abundant healthy scans.
Baselines
The naive reference point is the pooled counterfactual pipeline with no site awareness. The diffusion model and finite-difference mask extractor of Conditional Diffusion Models for Weakly Supervised Medical Image Segmentation are trained on all sites mixed together, and their masks are used directly and to train the same U-Net. Its reported BraTS result is 0.563 Dice. That number comes from a random split, so we reproduce it on our split before comparing. No reported result was found on Lacuna for leave-one-center-out weakly supervised BraTS segmentation, so the held-out-center figure must come from our own runs.
The strongest existing weak method is PathoGAN, which reports about 72.9% Dice on BraTS slices 60–100. We run it with the same slice range, sites, and tuning budget. To make the comparison fair, we also give it site-balanced sampling.
The strongest harmonization competitor is harmonize-then-segment. A CycleGAN maps every scan, including unseen-center scans, to a reference site, and the pooled diffusion pipeline is applied afterwards. This follows the input-space adaptation logic of Adapt the Data, Not the Model.
Two further baselines frame the imbalance and supervision questions. The standard imbalance treatment is the pooled pseudo-label U-Net trained with inverse-frequency center sampling, Dice plus cross-entropy loss, and strong intensity and bias-field augmentation. The supervision ceiling is a fully supervised U-Net, plus a Red-GAN-style site-conditioned synthesis that uses ground-truth masks. Both are labeled as using annotations we deliberately withhold.
Ablations isolate the key ingredient. One variant gives the diffusion model the site condition but never uses commutation. This tests whether conditioning alone helps. Another uses commutation without translated data, which separates the second stage from the third. A third replaces site translation with simple intensity transforms of the same strength, which tests whether learned site counterfactuals matter beyond generic photometric invariance.
All methods share the same backbone sizes, the same number of GPU-hours for generator training, and equal hyperparameter search budgets chosen on the validation centers only. The proposal beats a baseline if it improves Dice on the held-out center and on the rarest center by margins exceeding the spread across three seeds, while in-distribution Dice stays within one point.
Why not just…?
Why not just harmonize scanners first and run an existing counterfactual segmenter? This is the most serious threat. If an unpaired translator mapped every new scanner cleanly onto the training distribution, commutation would add little. The known weakness is that unsupervised translators also change pathological features while harmonizing texture, which causes missed lesions. Harmonize-then-segment can delete exactly the subtle lesions we care about, and nothing in the pipeline notices. Our approach uses the same kind of translation but checks it: a translation that moves the lesion map yields high $D$ and is discarded. The harmonization baseline settles this directly. If it matches us on held-out-center small-lesion Dice, the commutation idea is not needed and we will report that.
Why not just train on pooled pseudo-masks with heavy intensity augmentation and center-balanced sampling? This approach is cheap and often strong. The AMOS evidence of drops above 20% on unseen MRI scanners despite standard preprocessing suggests it is not enough for MRI. Augmentation also cannot repair pseudo-masks that are already wrong because the generator healed scanner texture. The intensity-transform ablation tests this: if generic photometric invariance matches learned site counterfactuals, the generative part of stage three is unjustified.
Why not just oversample or copy-paste lesions into rare-site scans? Duplicating scarce positives tends to memorize their idiosyncrasies (Tousignant et al.). Copy-paste moves a lesion with its source scanner's appearance into a foreign background, so the new image still looks like the source site. Site translation re-renders the whole slice coherently. We include copy-paste of pseudo-mask regions as an extra stage-three ablation.
Why not just prompt a medical foundation segmenter? MedSAM was outperformed by counterfactual attribution on vertebral metastases in Hide-and-Seek Attribution, and frozen BiomedCLIP features still produced 0% worst-group accuracy on ISIC in SpurCon. We still report box-prompted MedSAM as a reference. It uses information we do not, namely boxes.
Proposed Experiments
The primary benchmark is BraTS. We use its multi-institution site metadata, the same metadata Red-GAN used, and derive slice-level tumor/no-tumor tags from the masks, which are otherwise hidden during training. The protocol is leave-one-center-out. We also report a rare-center condition in which the rarest center, TCIA05, keeps only a handful of diseased training slices but all of its healthy ones.
A controlled benchmark answers the mechanism question directly. Synthetic anomalies with known masks, in the style of COIN's TotalSegmentator setup, are inserted into BraTS healthy slices from different centers. This shows whether failures under shift come from editing texture instead of the lesion.
A secondary benchmark is PolypGen's six colonoscopy centers. It is a much stronger shift between optical devices. It is usable only if enough lesion-free frames exist per center, and we will verify that before committing to it.
Metrics are Dice and 95th-percentile Hausdorff distance on the held-out center, worst-center Dice, rare-center Dice, and Dice stratified by lesion size. For the confidence score $D$, we report the rank correlation between $D$ and per-image Dice error.
Three checks decide the outcome. First, does commutation close a meaningful part of the in-distribution-to-held-out-center gap of the pooled diffusion baseline, while harmonize-then-segment does not? Second, does $D$ predict mask failure on unseen centers well enough to serve as a deployment-time flag? Third, on the controlled benchmark, does the pooled generator measurably edit background texture at foreign sites while the commuting generator does not? If all three come out negative, the scanner-confound explanation is wrong, and the field's weak-supervision gap lies elsewhere.
References
- COIN: Counterfactual Inpainting for Weakly Supervised Semantic Segmentation for Medical Images
- Pathology Segmentation Using Distributional Differences to Images of Healthy Origin
- Conditional Diffusion Models for Weakly Supervised Medical Image Segmentation
- Hide-and-Seek Attribution: Weakly Supervised Segmentation of Vertebral Metastases in CT
- Red-GAN: Attacking class imbalance via conditioned generation. Yet another medical imaging perspective
- AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation
- Unsupervised Structural-Counterfactual Generation under Domain Shift
- SpurCon: Weighted Supervised Contrastive Learning for Mitigating Spurious Cues in Medical Imaging
- Adapt the Data, Not the Model: Input-Space Adaptation for Frozen Time-Series Predictors