Models that rely on shortcuts fail when the shortcut breaks. A bird classifier that has learned "water background means waterbird" collapses on landbirds photographed near lakes. One clean fix keeps a pretrained network fixed, looks at its internal feature vectors, finds the directions that drift between training environments, and projects them away. This works well. On the Waterbirds benchmark it lifts worst-group accuracy of a ResNet-50 from about 63% to over 82% (Provable Domain Generalization via Invariant-Feature Subspace Recovery). But such a projection can only select among features the network still has. If ordinary training already threw away the weak, genuinely predictive cue (the bird's shape) because the strong background cue made the training loss small, no later projection can bring it back. We propose to split out-of-distribution failure into two separately measurable parts. The retention gap is invariant information missing from the features. The selection gap is invariant information present but not picked out. We then test whether a reconstruction objective, meaning a loss that forces features to be able to redraw the input, closes the retention gap while a provable projection step closes the selection gap.
The two-sentence version is this. Asking a network to reconstruct its input keeps information that label-driven training discards, including the weak features that survive distribution shift. It also keeps nuisance. So reconstruction should be used only to keep, and a separate environment-aware projection only to select. The reconstruction-discrimination trade-off then stops being a single weight to tune and becomes a knob that trades retention against how hard the selection problem is. We predict that worst-group accuracy after selection peaks at an intermediate reconstruction weight, and that the location of that peak can be explained by the two measured gaps.
Core Claim
Write $\phi_\lambda$ for a feature extractor trained with cross-entropy plus $\lambda$ times a reconstruction loss. Let $S$ be any post-hoc linear selection rule, such as projecting onto an invariant subspace. Let $\mathrm{err}_{\mathrm{wg}}$ denote worst-group test error. We decompose
$$\mathrm{err}_{\mathrm{wg}}(S \circ \phi_\lambda) = \underbrace{\mathrm{err}_{\mathrm{wg}}(\mathrm{oracle} \circ \phi_\lambda)}_{\text{retention gap}} + \underbrace{\left[\mathrm{err}_{\mathrm{wg}}(S \circ \phi_\lambda) - \mathrm{err}_{\mathrm{wg}}(\mathrm{oracle} \circ \phi_\lambda)\right]}_{\text{selection gap}},$$
Here "oracle" is a linear probe trained on group-balanced data that includes test groups. It gives the best a linear reader of $\phi_\lambda$ could do, and it is used only for analysis, never for model selection.
The claim has three parts. First, for ERM features ($\lambda = 0$) on shortcut-heavy data, the retention gap is a large share of the remaining error after invariant subspace recovery. Second, increasing $\lambda$ shrinks the retention gap monotonically while enlarging the selection gap, because more nuisance dimensions survive. Third, plain pixel mean-squared reconstruction is a poor retention signal. It spends capacity on high-energy content, such as backgrounds and staining, which is often exactly the spurious part. An energy-balanced reconstruction should therefore retain more invariant information per feature dimension. One option is to weight the loss on whitened targets. Another is a dense-plus-sparse decoder in which a dense, smooth component absorbs bulk energy and only the sparse residual carries most of the reconstruction weight.
Why this is not already done, and what constrains it
The motivation comes from the classical finding that energy is not relevance. Sparse Representation for Signal Classification combined a Fisher discriminant term with sparse reconstruction. On USPS with 5 dB noise it showed that this beats pure discrimination, at 20.6% versus 27.8% error. Low-Dimensional Density Ratio Estimation for Covariate Shift Correction argued that large-variance directions are often artifacts. Both papers stay within one distribution.
The selection method comes from ISR (Provable Domain Generalization via Invariant-Feature Subspace Recovery). ISR uses mean or covariance drift across two or more labelled environments to find the spurious subspace and keeps the orthogonal complement. It explicitly cannot recover features that the upstream model filtered out. That limitation is the gap we target.
One analysis constrains the design. In ISR's linear model, $x = A z_{\mathrm{inv}} + B z_{\mathrm{spu}} + \epsilon$ with isotropic noise. Here $z_{\mathrm{inv}}$ and $z_{\mathrm{spu}}$ are invariant and spurious latents mixed into $x$ by matrices $A$ and $B$. Explicitly explaining away the spurious part by joint reconstruction is an oblique decomposition of the kind analysed for dense-plus-sparse coding in Discriminative Reconstruction via Simultaneous Dense and Sparse Coding. It gives an estimate of $z_{\mathrm{inv}}$ with exactly the same signal-to-noise ratio as orthogonal projection. Orthogonal projection shrinks the signal by the sine of the principal angle between the two subspaces. The oblique solution amplifies noise by its inverse. The two effects cancel.
So bolting a decoder onto the selection step should give nothing in the linear regime. This rules out the most obvious combination of these directions. Any benefit from reconstruction must enter through retention in the learned representation, which is why we place it there.
The nearest existing work is CLEAR. CLEAR is a VAE that splits its latent into content and style and uses a supervised contrastive loss plus an anti-contrastive "pair-switching" loss to push label information out of style. It reports 0.747 (±0.019) accuracy on Camelyon17-WILDS. CLEAR asks a single network to reconstruct and to unlearn at once. When style and label are correlated, these two goals conflict, and its disentanglement has no recovery guarantee. We separate the two roles and give selection to a step with linear-model guarantees.
Supervised Representation Learning: Transfer Learning with Deep Autoencoders (TLDA) combines reconstruction, a softmax bottleneck and divergence alignment. It needs unlabeled target data and has no mechanism against spurious correlation, so it serves as our "retention without selection" reference. Task-Driven Dictionary Learning shows that supervision sharpens features in-distribution (0.54% versus 2.36% MNIST error). The same gradients that sharpen features will also chase shortcuts, which is the mechanism we expect to cause the retention gap at $\lambda = 0$.
Research Program
The work has three stages.
The first is a controlled linear-Gaussian study. Synthetic inputs mix a low-energy invariant latent and a high-energy spurious latent at controlled energy ratios, principal angles and environment counts. Under these conditions we can derive and check when supervised dimensionality reduction drops the invariant direction, and how much reconstruction weight restores it. A useful reference point is a linear autoencoder with a label head. Its retained subspace interpolates between the principal-component subspace and the discriminant subspace as $\lambda$ varies. Whether the invariant direction is kept follows from comparing its variance times $\lambda$ against its contribution to training loss. That comparison predicts the threshold $\lambda$ at which retention switches on.
The second stage trains ResNet-50 encoders on Colored-MNIST, Waterbirds and Camelyon17-WILDS over a grid of $\lambda$ values. We compare three reconstruction targets. The first is pixel mean-squared error. The second is reconstruction of whitened inputs. The third is a dense-plus-sparse decoder in the style of DenSaE, with reconstruction weight concentrated on the sparse part. For every encoder we compute the oracle probe, ISR-Cov and ISR-Mean, and plain supervised dimensionality reduction, which gives the two gaps.
The third stage is model selection without test groups. We choose $\lambda$ by worst-environment accuracy on held-out training environments after ISR. We then check whether this rule finds the peak that the oracle analysis reveals.
Baselines
The naive reference is ERM. We train a ResNet-50 with cross-entropy only and fit a linear probe. On Waterbirds, ISR reports roughly 63% worst-group accuracy for this setting. On Camelyon17-WILDS, WILDS reports about 93% in-distribution accuracy falling to about 70% out-of-distribution. We reproduce both numbers in our own pipeline before comparing anything.
The strongest existing competitor is ISR-Cov applied to ERM features. It is the only method in the file with a provable invariant-recovery step and the best reported Waterbirds result: over 82% worst-group accuracy with ResNet-50, and +18% over a linear probe on CLIP features. It also isolates exactly our variable, since our method is ISR-Cov applied to different features. To keep the comparison fair, it gets the same backbone, the same training epochs, the same environment labels, and the same budget of hyperparameter configurations. We give that budget, around twenty configurations, to the encoder search in both arms. Beating it means a worst-group accuracy gain on Waterbirds larger than the seed-to-seed standard deviation over five seeds. The gain must also be accompanied by a measured drop in the retention gap. A gain without that mechanism would not support the claim.
On Camelyon17-WILDS the reference is CLEAR-PS at 0.747 (±0.019). We rerun it with our backbone and budget. Group DRO is a further reference. It is reported only on CivilComments in the file (56.0% to 70.0% worst-group accuracy), and we found no reported result on Lacuna for Group DRO on Waterbirds, so we run it ourselves on Waterbirds with group labels.
Four ablations isolate the ingredients. The first is TLDA-style reconstruction plus softmax without the target-alignment term and without ISR, which tests retention without selection. The second is ISR at $\lambda = 0$, which tests selection without retention. The third swaps the reconstruction target from pixel mean-squared error to energy-balanced variants, which tests whether the kind of reconstruction matters. The fourth applies the oblique explain-away decoder at the selection stage. The linear analysis predicts that this last variant ties with orthogonal projection, so it acts as a check on our own theory.
Why not just…?
Why not just run ISR on CLIP features, which already retain far more than an ImageNet ERM model? This is the most serious threat. ISR's +18% on CLIP suggests large pretrained models may have a small retention gap already. We measure the CLIP retention gap directly with the oracle probe. If it is near zero on Waterbirds and Camelyon17, our method adds nothing for foundation-model users. The contribution would then shrink to the diagnostic and to settings without a suitable pretrained model, such as histopathology or microscopy domains. We would report this outcome plainly.
Why not just add an invariance penalty and train end to end? WILDS found that IRM and CORAL fail to beat ERM on 8 of 10 real-world datasets. Invariance penalties are also known to need many environments and to have fragile optimization. Our design keeps the invariance step convex and post hoc, with the two-environment guarantee of ISR-Cov.
Why not just use CLEAR-VAE, which already combines reconstruction with spurious-feature unlearning? CLEAR's reconstruction term actively resists unlearning when style and label are correlated, and its gains are heuristic. If our decoupled pipeline does not beat CLEAR-PS's 0.747 on Camelyon17 under matched compute, then the separation of roles is not buying anything. The Camelyon comparison is designed to settle this question.
Why not just tune the reconstruction weight in a TLDA-style autoencoder? Without a selection step, a supervised bottleneck aligns with whatever minimizes training loss, which includes the shortcut. The retention-only ablation tests this directly. We expect it to show a low retention gap but a large selection gap.
Decisive Checks
The proposal stands or falls on four observations. The first is Colored-MNIST, where colour carries high energy and is spurious, and digit shape carries low energy and is invariant. There, the retention gap at $\lambda = 0$ must be large and must fall with $\lambda$, and the fall must be faster under energy-balanced reconstruction than under pixel mean-squared error. The second is Waterbirds: worst-group accuracy after ISR must be non-monotone in $\lambda$, with a peak above ISR on ERM features. The third is the oblique-decoder ablation, which must tie with orthogonal projection; this confirms that the benefit comes from retention and not from the selection stage. The fourth is the model-selection rule, which must land within one seed standard deviation of the oracle-identified peak using only training environments. If the second check fails while the first succeeds, reconstruction keeps the invariant cue but in a form linear selection cannot isolate. That would point toward non-linear selection as the next problem.
References
- Provable Domain Generalization via Invariant-Feature Subspace Recovery
- Sparse Representation for Signal Classification
- Low-Dimensional Density Ratio Estimation for Covariate Shift Correction
- Discriminative reconstruction via simultaneous dense and sparse coding
- CLEAR: Unlearning Spurious Style-Content Associations with Contrastive LEarning with Anti-contrastive Regularization
- Supervised Representation Learning: Transfer Learning with Deep Autoencoders
- Task-Driven Dictionary Learning
- WILDS: A Benchmark of in-the-Wild Distribution Shifts