Some models look at a patient's baseline brain scan and predict how much that patient would benefit from a drug. The estimated quantity is the conditional average treatment effect (CATE): the expected difference between the outcome with treatment and the outcome without it, for patients who look like this one. These models could cut clinical trial sizes several-fold by enrolling likely responders. None of them has been tested on scans from a hospital or scanner it did not see during training. The standard fix for scanner shift in medical imaging is to train the network to ignore scanner "style" by randomizing intensity statistics. That fix was designed for diagnosis. It may also erase the faint image features that decide who responds to treatment. Those features are weaker than the ones that predict how sick a patient will get regardless of treatment, and they are easy to lose.
We propose two things. The first is a benchmark that pairs real multi-scanner brain MRI with simulated outcomes whose true treatment effects are known. On it, one can measure how much of a model's treatment-effect estimate survives a move to unseen scanners, as distinct from how much of its outcome prediction survives. The second is a training rule that asks for invariance where it matters. Under simulated scanner changes, the network is penalized only when its estimated treatment effect for a patient moves. It is not asked to make every internal feature scanner-invariant. The central claim is testable: robustness applied at the feature level trades away effect-modifier signal, and robustness applied to the output effect does not. The trade is worst when the true effect depends on small, localized anatomy.
What exists and what is missing
Three papers estimate treatment effects directly from images, and all of them evaluate only on data drawn from the training distribution. Personalized Prediction of Future Lesion Activity and Treatment Effect in Multiple Sclerosis from Baseline MRI trains a TARNet on 1,817 patients from four pooled multiple sclerosis trials. A TARNet is a network with a shared image encoder and one prediction head per treatment arm; here the encoder is a 3D ResNet. The authors show that grouping patients by predicted benefit separates strong responders from weak ones. They list generalization to unseen sites as an open limitation. Enhancing Predictive Imaging Biomarker Discovery Through Treatment Effect Analysis shows that a two-headed network finds predictive image features (those that change the treatment effect) where a single-headed network latches onto prognostic ones (those that predict the outcome under any treatment). It tests this on ISIC 2018 and NSCLC CT with simulated outcomes, but only in-distribution. Image-based Treatment Effect Heterogeneity groups satellite images into a small number of discrete response types and never faces sensor calibration shift. These papers motivate the question and supply the architecture we start from.
On the domain-shift side, Higher-Order Domain Generalization in Magnetic Resonance-Based Assessment of Alzheimer's Disease introduces Extended MixStyle (EM). EM blends skewness and kurtosis of intermediate features between random training pairs so that a network trained only on NACC generalizes to ADNI, AIBL and OASIS. It evaluates only on diagnosis. EM is both the method we reuse and the design we expect to fail in a specific way. Experience with Single Domain Generalization in Real World Medical Imaging Deployments reports that generic single-site generalization methods degrade on rare, fine-grained findings. That is the constraint behind our worry. Stable Heterogeneous Treatment Effect Estimation across Out-of-Distribution Populations makes treatment-effect estimation robust to covariate shift through reweighting and feature decorrelation, but only on tabular benchmarks. It also reports that decorrelation costs in-distribution accuracy. Adjustment for Confounding using Pre-Trained Representations shows that neural-network nuisance models inside double machine learning handle rotated pretrained embeddings, where Lasso-style estimators break. Double machine learning is an estimator that fits outcome and treatment-assignment models separately and removes their errors from the effect estimate. That paper also leaves conditional effects and shifting embedding geometry open.
The most obvious combination is a TARNet trained with EM and tested on new scanners. Nobody has run it. It is too obvious to be the contribution, so it becomes our main ablation. The new part is the effect-level evaluation and the effect-level invariance target.
Core Claim
In a randomized trial, treatment assignment is independent of the scanner. The true effect as a function of anatomy therefore does not change across sites. Only the encoder's reading of the anatomy changes. It follows that the quantity to hold fixed under a scanner change is the patient's estimated effect $\hat\tau(x)=\hat\mu_1(x)-\hat\mu_0(x)$, where $\hat\mu_w$ is the prediction head for arm $w$. Feature statistics as a whole do not need to be fixed. Forcing feature-level invariance, as EM and HSIC decorrelation do, removes variation the effect heads may need. Output-level consistency leaves the encoder free to keep scanner-correlated but biologically real detail, provided that detail does not move the effect estimate. We predict the gap between the two approaches grows as the true effect modifier becomes smaller and more local.
Research Program
The model is a TARNet with a 3D ResNet encoder, trained on a single source site. During training we draw acquisition perturbations $g$ from a physics-motivated family applied in input space: bias fields, contrast and gamma curves, blur and resolution changes, noise, and moment perturbations in the spirit of EM. The training objective is
$$\mathcal{L}=\sum_i \ell\big(y_i,\hat\mu_{w_i}(x_i)\big)+\ell\big(y_i,\hat\mu_{w_i}(g(x_i))\big)+\lambda\,\big(\hat\tau(g(x_i))-\mathrm{sg}[\hat\tau(x_i)]\big)^2 .$$
Here $y_i$ and $w_i$ are patient $i$'s outcome and treatment, $\ell$ is the outcome loss, $\mathrm{sg}$ stops gradients through the clean-image estimate, and $\lambda$ sets the strength of the consistency penalty. The factual terms keep $\hat\tau$ from collapsing to a constant.
Choosing hyperparameters is itself a problem. Individual effects are never observed, and diagnostic accuracy is the wrong selection criterion. We select $\lambda$ and the augmentation strength with a doubly robust pseudo-outcome loss computed on perturbed held-out source images. A pseudo-outcome is a per-patient quantity whose expectation equals the true effect. We also report what happens when an oracle selects using the true effects, so the cost of realistic model selection is visible.
The benchmark is the second product. We train on NACC (n=4,647 T1 scans) and test on ADNI, AIBL and OASIS (n=3,126). These are the cohorts and scanner mix EM used, so the acquisition shift is real. Treatment is simulated as randomized, with a confounded variant added later. Outcomes follow $Y=\mu_0(x)+W\tau(x)+\varepsilon$, where $W$ is the treatment indicator and $\varepsilon$ is noise. The baseline outcome $\mu_0$ is driven by strong, global prognostic quantities such as ventricular volume and age. The effect $\tau$ is driven by one of three families of modifiers. The first is a global volume such as hippocampal volume. The second is a small, localized regional thickness. The third is a contrast-based quantity, such as a gray/white intensity ratio, that is partly biological and partly scanner-dependent. All modifiers are computed from each scan with one standardized segmentation pipeline. This treats measured anatomy as ground truth. The pipeline itself has some scanner sensitivity, which we quantify with test-retest-style comparisons where cohorts permit. The third family is a deliberate stress test, because "invariance" there is ambiguous. We borrow the outcome dynamics and the knobs for sample size, overlap and effect heterogeneity from ADCB: An Alzheimer's disease benchmark for evaluating observational estimators of causal effects, which simulates progression from ADNI parameters. Unlike ADCB, we tie the effect to image-derived anatomy and evaluate across scanners.
Beyond aggregate error, we split each model's error under the site change into the part caused by drift in $\hat\mu_0$ and the part caused by drift in $\hat\tau$. We also report an effect retention ratio: the rank correlation between estimated and true effects on target sites, divided by the same correlation on the source site. A validation on real data is possible only if trial imaging access can be arranged with the multiple sclerosis consortium behind the TARNet study. In that case, holding out whole trials or sites and comparing responder-bin uplift and Qini curves (a measure of how well a ranking concentrates benefit in the top-ranked patients) would test the claim without simulated outcomes. We treat this as contingent, not promised.
Baselines
The naive reference point is a plain TARNet trained on NACC with ordinary augmentation and applied unchanged to ADNI, AIBL and OASIS, following the MS TARNet design. It tells us whether scanner shift degrades effect estimation at all. If it does not, the project's premise fails and we report that.
The strongest existing domain-generalization competitor is the same TARNet with EM inserted in its encoder, as in Higher-Order Domain Generalization in Magnetic Resonance-Based Assessment of Alzheimer's Disease. On diagnosis, EM reported improving out-of-domain macro-F1 by 2.4 percentage points on average over earlier single-site generalization methods on this exact NACC-to-ADNI/AIBL/OASIS split. No reported result was found on Lacuna for any treatment-effect metric under scanner shift, so every effect-level number in this study will be new. EM is run with the same encoder, data, epochs and hyperparameter search budget as our method. Its mixing strength is selected with the same pseudo-outcome criterion, so it is not handicapped by accuracy-based selection.
The strongest tabular competitor is a causal forest and a doubly robust learner fitted on standardized volumetric and thickness measurements plus age and sex. This mirrors the scalar-feature pipeline of Estimating individual treatment effect on disability progression in multiple sclerosis using deep learning. On real trial data, that pipeline reached a hazard ratio of 0.361 in its top 30% of predicted responders, against 0.743 overall, and implied a trial of 245 rather than 1,374 patients. We also run a doubly robust learner with neural nuisance models on frozen pretrained image embeddings, extending the approach of Adjustment for Confounding using Pre-Trained Representations from average to conditional effects. SBRL-HAP, applied to the same embeddings, is the tabular out-of-distribution representative.
Three ablations isolate the key ingredient. The first applies consistency to features (matching encoder embeddings across $x$ and $g(x)$). The second applies consistency to $\hat\mu_0$ and $\hat\mu_1$ separately. The third applies consistency to $\hat\tau$ alone. All use the identical perturbation family. We count the proposal as winning if the third ablation beats EM-TARNet on target-site effect error (PEHE, the root-mean-squared error of $\hat\tau$) and on the effect retention ratio for the localized-modifier family. The margin must exceed seed-to-seed variation over five seeds on at least two of the three target cohorts, and source-site PEHE must stay close to that of plain TARNet.
Why not just…?
Why not just extract standardized volumes and use a causal forest? This is a real threat. When the true modifier is a global volume that the pipeline measures well, we expect it to match or beat every image model, and the benchmark's first modifier family exists to show exactly that. Pixel-level models should win only when the modifier is not among the pre-specified measurements. That is the situation the MS TARNet study reports on real data, where adding raw MRI improved responder separation over clinical variables. Our benchmark checks this claim under scanner shift, which that study did not. If the forest also wins on the localized family, the honest conclusion is that end-to-end image models do not currently earn their added scanner fragility, and we will say so.
Why not just add EM or heavy augmentation to TARNet? This is the main ablation, not a dismissed alternative. The biomarker paper shows predictive signal is easily dominated by prognostic signal, and the deployment study shows generic generalization blurs fine structure. Together they predict EM will keep diagnostic-like prognosis while losing localized effect modifiers. The error split into $\hat\mu_0$ drift and $\hat\tau$ drift tests this directly. If EM-TARNet matches output-level consistency on the localized family, our central claim is wrong.
Why not just harmonize target scans into source style with an image translator and keep the model frozen? Translation approaches such as the 3D CycleGAN in UNSUPERVISED HARMONIZATION OF BRAIN MRI USING 3D CYCLE GANS AND ITS EFFECT ON BRAIN AGE PREDICTION, or input adapters like Adapt the Data, Not the Model: Input-Space Adaptation for Frozen Time-Series Predictors, need unlabeled target-site scans. Our single-source setting forbids that. We still run CycleGAN harmonization as a privileged baseline with target access. If it beats our method even so, that tells practitioners to collect target scans before deployment. We also measure whether translation alters the modifier measurements themselves, which would silently corrupt the effect.
Why not just feed image embeddings to an out-of-distribution tabular estimator like SBRL-HAP? Its HSIC decorrelation acts on embedding coordinates. The pretrained-representation paper shows that coordinate-based assumptions such as sparsity break under the arbitrary rotations of neural embeddings. SBRL-HAP's own authors also report an in-distribution cost from decorrelation. We run it rather than rely on this argument. Because it operates on frozen features, it also serves as a check on whether the robustness gain requires training the encoder at all.
Decisive Checks
The first check is whether plain TARNet's effect retention ratio falls well below one on unseen cohorts while its outcome prediction degrades much less. That would establish that scanner shift is a larger problem for effect estimates than for prognosis. The second is whether EM-TARNet recovers outcome accuracy but not retention on the localized-modifier family, while output-level consistency recovers both. The third is whether the ordering flips on the global-volume family, where the tabular forest should be competitive. The fourth is how much the pseudo-outcome selection criterion loses relative to oracle selection. If that loss is large, choosing models without ground-truth effects is the real bottleneck, and it deserves its own study. For enrichment, we report the true effect in the top 30% of patients ranked by $\hat\tau$ on each target cohort. This mirrors how the MS enrichment study presented its clinical case and ties the benchmark to the trial-size question that motivates the work.
References
- Personalized Prediction of Future Lesion Activity and Treatment Effect in Multiple Sclerosis from Baseline MRI
- Enhancing Predictive Imaging Biomarker Discovery Through Treatment Effect Analysis
- Image-based Treatment Effect Heterogeneity
- Higher-Order Domain Generalization in Magnetic Resonance-Based Assessment of Alzheimer's Disease
- Experience with Single Domain Generalization in Real World Medical Imaging Deployments
- Stable Heterogeneous Treatment Effect Estimation across Out-of-Distribution Populations
- Adjustment for Confounding using Pre-Trained Representations
- ADCB: An Alzheimer's disease benchmark for evaluating observational estimators of causal effects
- Estimating individual treatment effect on disability progression in multiple sclerosis using deep learning
- UNSUPERVISED HARMONIZATION OF BRAIN MRI USING 3D CYCLE GANS AND ITS EFFECT ON BRAIN AGE PREDICTION
- Adapt the Data, Not the Model: Input-Space Adaptation for Frozen Time-Series Predictors