When a diagnostic model is trained on images pooled from several hospitals, each hospital pulls the model's weights in its own direction. One family of methods, cross-domain gradient alignment, exploits this. It keeps only the parts of each weight update that every hospital agrees on. The bet is that real pathology helps everywhere, while a hospital-specific artifact, such as a scanner watermark, helps in one place and hurts in another. The trouble is that hospitals differ in more than their scanners. A cancer referral center may see five times the disease prevalence of a screening clinic. Each hospital's update, averaged over a batch, therefore mixes two signals: "which image features matter" and "how often is the answer positive here." Forcing agreement on these blended updates can throw away useful, class-specific learning or keep shortcuts that happen to line up with base rates.
We propose to compute agreement separately within each diagnostic class. For each update, we ask whether the hospitals agree on how to change the model for their positive cases, and separately whether they agree for their negative cases. Only coordinates with within-class agreement across hospitals are applied. The method needs hospital identifiers, which every multi-center archive records. It does not need per-image confounder labels such as sex, device presence, or rulers, which archives rarely contain. A controlled study is built in to test the method's core assumption: alignment can only remove a shortcut whose link to the label differs between hospitals.
Core Claim
Our claim has two parts. First, under realistic label-prevalence shift between hospitals, existing agreement-based methods partly align base-rate differences rather than features. Restricting agreement to within-class, across-hospital comparisons removes that confound and recovers shortcut robustness. Second, this gain exists only when the shortcut's predictive strength varies across hospitals. When a shortcut is uniform across sites, any method that sees only site labels should fail, and we will show that it does rather than hide it.
Write $g_{k,c} = \nabla_\theta \mathcal{L}_{k,c}(\theta)$ for the gradient of the loss on examples of class $c$ from hospital $k$, with $K$ hospitals. For each class we form a coordinate-wise agreement score: the absolute mean of the signs across hospitals,
$$a_{c,j} = \Big|\tfrac{1}{K}\textstyle\sum_{k} \operatorname{sgn}(g_{k,c,j})\Big|,$$
which equals 1 when all hospitals agree on coordinate $j$ for class $c$. The update is then
$$\Delta\theta_j = -\eta \sum_{c} \mathbb{1}[a_{c,j} \ge \tau]\;\tfrac{1}{K}\textstyle\sum_k g_{k,c,j}.$$
Here $\eta$ is the learning rate and $\tau$ is an agreement threshold. Setting $\tau=1$ recovers strict agreement and smaller values give a softer filter.
This rule contrasts with three existing methods. Agr-Sum from Domain Generalization via Gradient Surgery applies sign agreement to each domain's whole-batch gradient. We reuse its masking idea but change what is compared. Fishr (Fishr) matches per-domain gradient variances in the classifier layer, and those variances are also mixtures over each domain's class proportions. Common Gradient Descent (Focus on the Common Good) seeks updates that agree with every annotated group. If its groups are site×class cells, it still asks positive-class and negative-class gradients to agree, even though they legitimately oppose each other at the decision boundary. Our rule never compares across classes.
Research Program
The work has three parts.
Measurement. We first measure the problem before fixing it. On real multi-center data, we track per-hospital whole-batch gradient cosines alongside per-class cross-hospital cosines throughout training. The aim is to test the observation of Mansilla et al. that within-domain gradients agree more than cross-domain ones. We then ask how much of the cross-domain conflict disappears once class is held fixed. If most of it disappears, prevalence was masquerading as domain conflict and the method is well-motivated. If little disappears, the second part of the program is weakened, and we will report that.
Method. The second part is the method itself. It is applied, following Fishr's practice for tractability, to the classifier head and the last backbone block after a short ERM warmup. Batches are stratified so that every hospital-by-class cell appears. For cells too small to give a stable gradient, we use an exponential moving average of that cell's past gradients. We also test the same filter on a frozen pretrained chest-X-ray encoder with a trainable head. This setting matters because regulators may require validated backbones to stay fixed, a constraint motivating INPUTADAPTER.
Controlled shortcut study. The third part is a controlled study that tests the mechanism rather than only the leaderboard. The study builds on CheXpert pleural effusion detection with a sex confounder, following Benchmarking Dependence Measures to Prevent Shortcut Learning in Medical Imaging. We split training data into $K$ synthetic "hospitals" and control two quantities independently: how much effusion prevalence differs across hospitals, and how much the sex–effusion correlation differs across hospitals. Evaluation is on the inverted test set used in that paper. This grid lets us separate the two effects the core claim depends on.
Baselines
Pooled ERM. The naive reference point is pooled ERM with class-balanced sampling, the cheapest thing a hospital would actually do. On Camelyon17-WILDS, WILDS reports ERM falling from about 93% in-distribution accuracy to about 70% on the held-out hospital. On the inverted sex-confounded CheXpert task, Müller et al. report a baseline AUROC of 45.8. Beating ERM is necessary but not interesting.
Global agreement methods. The strongest competitors that use the same supervision we use, hospital labels only, are Agr-Sum and Fishr. They are closest in mechanism and need no confounder labels. Fishr is reported in its paper as the only DomainBed method to beat ERM on all five real-world datasets, at 0.2% extra memory and 2.7% extra time on PACS. We found no reported result on Lacuna for Fishr or Agr-Sum on Camelyon17, CheXpert, or ISIC, so we will run them ourselves. They get the same backbone, warmup, alignment layers, data splits, and an equal hyperparameter search budget over learning rate and regularization strength, selected by validation on training hospitals only. Matching or beating them only in the zero-prevalence-shift corner of the controlled grid would not count as success. Success means a gap that widens as prevalence shift grows, with confidence intervals over five seeds that exclude zero.
CGD with site×class groups. We run CGD with groups defined as site×class cells. This gives it exactly the information our method uses and isolates the difference between agreeing within a class and agreeing across all groups.
Confounder-supervised references. MINE (AUROC 87.7 on inverted CheXpert, per Müller et al.) and MIMM-X (70.4% accuracy on its inverted multi-confounder CheXpert set) use confounder labels we deliberately do not use. They serve as upper references rather than competitors. Closing a substantial fraction of the gap to MINE without sex labels would be a meaningful result. Exceeding it is not expected.
Ablations. The key ablation keeps everything identical, including the stratified site×class batching, and switches only the agreement test from per-class to whole-batch. This isolates the class-conditioning as the active ingredient.
Why not just…?
Why not just rebalance each hospital to equal class prevalence and run Agr-Sum or Fishr? This is the most serious threat, because rebalancing removes the base-rate component from each hospital's gradient. It does not remove the masking problem, though. A balanced hospital gradient is still the sum of a positive-class and a negative-class gradient. A cross-hospital conflict on positives, for example one site rewarding a device artifact, can be hidden by strong agreement on negatives. Whether this masking matters in practice is an empirical question. The key ablation, run with balanced batches for both arms, settles it directly. If the two arms tie across the controlled grid, we will conclude that rebalancing suffices and report it. Müller et al. did find that rebalancing alone "failed to 'clean' the latent space," but that was rebalancing without alignment, so it does not settle this question.
Why not just use Group-DRO or CGD with site×class groups? Piratla et al. show Group-DRO overfitting to noisy minority groups: 48.5% versus CGD's 65.6% worst-group accuracy on Colored MNIST. Medical minority cells such as positives from a small hospital are exactly where label noise concentrates, which counts against Group-DRO. CGD is the closer competitor. Its objective rewards agreement between opposing-class groups, which we expect to slow learning of the decision boundary. We expect this to show up as lower in-distribution AUROC at matched worst-group performance. CGD is run as a baseline under the same budget.
Why not just use the confounder-label methods, or SpurCon's few-shot labeling? SpurCon reaches 73.0% worst-group accuracy on CheXpert pneumothorax and 65.0% on ISIC 2020 with about ten expert labels per artifact group. When a clinician already knows the shortcut, that is a strong and cheap option. Our method targets the complementary case of shortcuts nobody has named, using only hospital identity. We will report SpurCon's published numbers as a reference and run it on the ISIC setting where we can reproduce its protocol. We will not claim superiority where the shortcut is known.
Proposed Experiments
Controlled CheXpert grid. This is the decisive test. It crosses prevalence shift (none, moderate, severe) with shortcut heterogeneity across sites (uniform, moderate, strong). It reports inverted-test AUROC and worst-group accuracy for every method in the Baselines section. The first prediction is that, at strong shortcut heterogeneity, global Agr-Sum and Fishr lose ground as prevalence shift grows while the class-conditional rule stays roughly flat. The second prediction is that, at uniform heterogeneity, every site-label method, ours included, stays near the ERM baseline. A gain in that column would mean our method is doing something other than what we claim, and we would investigate before reporting.
Real hospital shift. Leave-one-hospital-out evaluation on Camelyon17-WILDS uses out-of-distribution accuracy, referenced against ERM's roughly 70%. The 94.9% StarAlign figure from FedPDA uses target labels and is not comparable. Leave-one-source-out evaluation on the HAM10000/BCN20000/MSK/D7P skin-lesion union, whose sources differ in class mix, uses weighted accuracy and worst-class recall. These two benchmarks test whether the controlled finding survives real, unmanipulated prevalence and acquisition shift.
Diagnostics. Every run logs the within-class versus whole-batch cross-hospital gradient cosines from the measurement stage. Every run also includes a linear probe predicting sex from the final features on CheXpert, mirroring MIMM-X's latent-scrubbing check, to test whether robustness comes from removing confounder information rather than from incidental regularization.
References
- Domain Generalization via Gradient Surgery
- Fishr: Invariant Gradient Variances for Out-of-Distribution Generalization
- Focus on the Common Good: Group Distributional Robustness Follows
- Adapt the Data, Not the Model: Input-Space Adaptation for Frozen Time-Series Predictors
- Benchmarking Dependence Measures to Prevent Shortcut Learning in Medical Imaging
- WILDS: A Benchmark of in-the-Wild Distribution Shifts
- MIMM-X: Disentangling Spurious Correlations for Medical Image Analysis
- SpurCon: Weighted Supervised Contrastive Learning for Mitigating Spurious Cues in Medical Imaging
- Post-Deployment Adaptation with Access to Source Data via Federated Learning and Source-Target Remote Gradient Alignment