Medical image classifiers often learn shortcuts: easy signals, such as a chest tube placed to treat a collapsed lung or the patient's sex, that happen to correlate with the diagnosis in training data but are not the disease itself. Every mitigation method in the literature is built and tested against one shortcut at a time. Real hospital data contains several at once. This note asks whether fixes for different shortcuts add up when combined, or whether they interfere. One kind of interference is that removing one shortcut pushes the model onto another. A second is that two debiasing objectives trained together degrade each other. We propose to measure this interaction directly on chest X-ray benchmarks where two shortcuts can be controlled independently.
We then test a way to combine fixes that borrows from model merging, the practice of adding together weight changes from separately fine-tuned copies of one network. For each shortcut, the "debias vector" is the weight difference between a model fine-tuned with a debiasing objective and one fine-tuned without it. We ask two questions. First, can sparse debias vectors, each computed for a single shortcut, be summed into one model that is robust to both? Second, does the geometry of these vectors, such as their overlap, predict ahead of time whether two fixes will help or hurt each other? A clear answer either way matters. If fixes compose, hospitals could add a new shortcut fix without retraining. If they do not, single-shortcut robustness results overstate how robust deployed models are.
Core Claim
The central claim is that shortcut mitigations interact non-additively, and that the interaction can be measured and partly predicted. Let $R$ be a robust metric, such as worst-group accuracy over all disease-by-confounder subgroups. Let $R_0$ be its value for plain training, $R_A$ and $R_B$ its values when only shortcut $A$ or only shortcut $B$ is fixed, and $R_{AB}$ its value when both are fixed. The interaction is then
$$I_{AB} = (R_{AB} - R_0) - (R_A - R_0) - (R_B - R_0).$$
Zero means the fixes are additive. Negative means the fixes interfere, and positive means they reinforce each other. We expect a negative $I_{AB}$ when a spatially local fix, such as masking chest tubes, is paired with a diffuse confounder like sex. Removing the easy local cue should raise reliance on the remaining diffuse one.
The method claim is narrower and more easily falsified. Debias vectors made sparse with the dual-tail masking of Model Breadcrumbs should compose better than naive summation. The measured $I_{AB}$ should track the overlap of the two vectors in weight space. That would give an a priori test for whether two modules will interfere, which the synergistic-integration literature lists as an open problem.
Why This Gap Is Real
The shortcut literature supplies single-confounder methods, and each is evaluated on one confounder. Benchmarking Dependence Measures to Prevent Shortcut Learning in Medical Imaging splits the latent representation into disease and confounder parts and penalizes dependence between them. It does this for sex versus pleural effusion alone. Detecting shortcut learning for fair medical AI using shortcut testing varies gradient reversal on one attribute head to test whether that attribute is actually used. It also shows that encoding an attribute is not the same as relying on it, which constrains us: we must measure reliance, not just encoding. SpurCon targets support devices on CheXpert and rulers on ISIC with a frozen backbone and a reweighted contrastive head. Localized Shortcut Removal in-paints localized artifacts and states that it cannot handle diffuse ones. Reducing Reliance on Spurious Features … with Spatial Specificity uses segmentation supervision against chest drains. None of these papers asks what happens to a second shortcut when the first is removed.
Hidden stratification, as documented by Oakden-Rayner et al., is the clinical reason this matters. A fix that closes one subgroup gap while opening another can leave aggregate metrics unchanged.
The integration literature supplies the lens and the tool. Work such as FLoMo-Net and the multi-scale contour-loss segmentation study reports that modules combine non-additively, but only for gains on in-distribution benchmarks. We apply the same interaction accounting to robustness under distribution shift. From the merging work we reuse two techniques: the sparse-mask method of Model Breadcrumbs and, as a stronger variant, the shared-plus-routed decomposition of Twin-Merging. Both were shown on multi-task NLP and generic vision, never on debiasing objectives. Merging requires a shared base architecture, and that constraint suits our setting, because every debiased copy starts from the same pretrained chest X-ray backbone.
Research Program
The first step is a factorial benchmark with two independently controllable shortcuts.
On CheXpert, we subsample to correlate pleural effusion with patient sex, following the dependence-measures benchmark. At the same time, we correlate the pneumothorax or effusion label with the support-device label. This creates eight disease-by-sex-by-device groups. We build test splits that invert only the sex correlation, only the device correlation, or both. The singly inverted splits isolate reliance on each shortcut.
On NIH-CXR14, we replicate the setup with age, following ShorT, and with chest tubes, using the NEATX tube annotations shipped in the clip_cxr_fairness codebase. We vary correlation strength on a grid so that interaction can be studied as a function of how strong each shortcut is. As a sanity check, Morpho-MNIST gets a second synthetic confounder added alongside stroke thickness.
The second step estimates $I_{AB}$ for several pairs of fixes. The pairs mix method families: a dependence penalty against sex paired with SpurCon-style reweighting against devices, a Lens-style in-painting front end for tubes paired with gradient reversal for age, and the same method family applied to both shortcuts. Each pair is combined three ways: joint training, sequential fine-tuning, and weight-space merging.
The third step computes per-shortcut debias vectors. Every model starts from one pretrained backbone, taken from the DINOv2 radiology release or CXR-CLIP. For each shortcut, the debias vector is the weights of a copy fine-tuned with the debiasing objective minus the weights of a copy fine-tuned identically but without it. These vectors are sparsified with dual-tail masks and summed with a scaling coefficient tuned on a validation split. We record the vectors' cosine similarity and the overlap of their masked support. We then test whether these quantities predict $I_{AB}$ across all pairs and correlation strengths.
Reliance is measured two ways: by performance on the singly inverted splits, and by ShorT-style gradient-scaling sweeps. This keeps "the model encodes sex" separate from "the model uses sex."
Baselines
The naive reference is standard fine-tuning of the same backbone on the biased training set, with no debiasing (empirical risk minimization, or ERM). The dependence-measures benchmark reports that this kind of baseline reaches only 45.8 AUROC on the sex-inverted CheXpert split. We expect a similar collapse on our splits.
The strongest competitor for handling both shortcuts at once is joint training with both debiasing penalties active. This is the MINE dependence penalty from Benchmarking Dependence Measures, run with two confounder subspaces instead of one. It is strongest because it optimizes both objectives together with full access to both labels. In its single-confounder form it reports 87.7 AUROC on the sex-inverted split.
For devices alone, the strongest reported method is SpurCon, which reports worst-group accuracy rising from 35.6% to 73.0% on CheXpert support devices. No result for simultaneous multiple-shortcut mitigation was found on Lacuna, so the joint two-penalty model is our own reproduction.
A second strong competitor is group-balanced reweighting over all eight intersectional groups. It uses exactly the same labels as our method and is cheap to run.
Fairness rules apply to every arm. All arms use the same backbone initialization, the same training images, and the same confounder labels. Hyperparameter search budgets are matched in GPU-hours, and the merging coefficient counts toward that budget. Model selection uses the same validation split, which is balanced over the eight groups. Results are averaged over five seeds and reported with confidence intervals.
Merging beats joint training only if it achieves worst-group accuracy on the doubly inverted split within confidence of the joint model, or above it, at lower cost. It also counts as a win if merging lets a third shortcut be added without retraining while holding worst-group accuracy on the first two.
Two ablations isolate the key ingredient. Plain task-arithmetic summation, without sparse masks, tests whether masking matters. Sequential fine-tuning, fixing one shortcut and then the other, tests whether weight-space composition differs from simply stacking training runs.
Why not just…?
Why not just train one model with every penalty turned on? This is the most serious threat, and it is the main baseline, not a straw man. If joint training matches merged debias vectors on every split, the method claim fails. The interaction measurements still stand: they would show whether single-shortcut robustness numbers survive a second confounder. Joint training does carry two known costs. It requires retraining whenever a new shortcut is discovered, and the dependence-measures benchmark notes that the auxiliary MINE estimator is unstable and expensive. The add-a-third-shortcut experiment tests whether merging's modularity is worth anything in practice.
Why not just rebalance over the intersectional groups? Rebalancing is cheap and uses identical labels, so it could plausibly win on worst-group accuracy. However, the dependence-measures benchmark found that rebalancing raised accuracy without removing confounder information from the representation. With eight groups, some cells in a strongly correlated split will be tiny. We report both accuracy and ShorT-style reliance for rebalancing. If rebalancing wins on both, we say so.
Why not just chain existing tools, such as a Lens in-painter for tubes followed by a dependence penalty for sex? This pipeline is one of our composition arms. Localized Shortcut Removal explicitly cannot address diffuse cues. Our hypothesis predicts that erasing the tube will raise reliance on sex, which is exactly a negative $I_{AB}$. Whether the downstream penalty absorbs that shift is an empirical question the factorial splits answer directly.
Proposed Experiments and Decisive Checks
The first decisive check is the sign of $I_{AB}$. For the device-plus-sex pair on CheXpert, we test whether the doubly inverted worst-group accuracy of the jointly fixed model falls significantly below the additive prediction. We also test whether fixing devices alone lowers AUROC on the sex-inverted split relative to ERM. A consistently zero interaction across all pairs and correlation strengths would falsify the premise, and we would report that single-shortcut benchmarks are adequate.
The second check is composition. Sparse-masked merged debias vectors are compared against joint training, rebalancing, and plain summation on all three inverted splits. The comparison is repeated on NIH-CXR14 with age and tubes to test whether findings transfer across datasets.
The third check is prediction. We compute the rank correlation between debias-vector overlap and measured $I_{AB}$ across roughly two dozen pair-by-strength configurations, using cross-validation over pairs so that no configuration is used to predict itself.
The fourth check relaxes the label requirement. Confounder labels are replaced with SpurCon's few-shot prototype pseudo-labels, and we measure how much each composition method degrades. This matters because gold labels for every shortcut are rarely available in practice.
Any robustness gain must hold across seeds and must not cost more than a few points of in-distribution AUROC. The accuracy penalty reported for eXBL shows that this trade-off decides whether clinicians adopt a fix.
References
- Model Breadcrumbs: Scaling Multi-task Model Merging with Sparse Masks
- Benchmarking Dependence Measures to Prevent Shortcut Learning in Medical Imaging
- Detecting shortcut learning for fair medical AI using shortcut testing
- SpurCon: Weighted Supervised Contrastive Learning for Mitigating Spurious Cues in Medical Imaging
- Localized Shortcut Removal
- Reducing Reliance on Spurious Features in Medical Image Classification with Spatial Specificity
- Hidden Stratification Causes Clinically Meaningful Failures in Machine Learning for Medical Imaging
- FLoMo-Net: A Novel Task-Adaptive Mixture of Experts Routing Framework with Frequency and Uncertainty Correction for Medical Image Segmentation
- A Medical Image Segmentation Method based on Multi-scale Features and Contour Loss Constrain (S)
- Twin-Merging: Dynamic Integration of Modular Expertise in Model Merging
- Unlearning Spurious Correlations in Chest X-Ray Classification