Self-supervised vision models learn from unlabeled images by solving a made-up task, for example making two augmented copies of the same image map to nearby feature vectors. They often solve that task the cheap way. They match background color, texture, compression noise or lens artifacts instead of objects. A popular family of methods called feature whitening forces every dimension of the learned representation to stay active and uncorrelated with the others. This is widely credited with stopping "collapse," the failure where features shrink into a few directions. We argue that whitening prevents a geometric failure but not an informational one. A cheap shortcut can carry enough distinct information to fill every whitened dimension by itself. A whitened model can therefore be perfectly full-rank and still blind to the objects in the image.
We propose a simple, label-free fix and a set of tests that would show whether it works. Alongside the main encoder we train a deliberately weak encoder that can only see small local patches, so it can only learn low-level statistics. The whitening objective is then applied to the part of the main representation that the weak encoder's features cannot predict. If this works, it would give a cheap alternative to the pixel-reconstruction branches currently used against "feature suppression," where easy features crowd out harder ones. It would also give a direct account of when whitening helps close the gap to supervised features and when it cannot.
Core Claim
Whitening constrains the second-order geometry of the features: the full matrix of variances and covariances between dimensions. It does not constrain where the information comes from. Take a shortcut made of $k$ shared random bits, as in the RandBit benchmark of Intriguing Properties of Contrastive Losses. Those bits admit up to $2^k-1$ mutually uncorrelated, unit-variance functions (the Walsh basis). With $k=10$, that means any embedding of up to 1023 dimensions can be exactly whitened and exactly invariant across the two views while encoding nothing but the bits. A continuous shortcut such as background hue offers unlimited capacity. Whitening-based objectives therefore have shortcut solutions that are global optima, not merely bad local minima.
Our first claim is that W-MSE, CW-RGP and INTL will be about as vulnerable as SimCLR on shortcut benchmarks. Our second claim is that the remedy is to make the decorrelation conditional on a model of the shortcut. That model must be learned without bias labels, and it must be cheap compared with inpainting.
Concretely, let $z_1, z_2$ be the main encoder's embeddings of two views and $s_1, s_2$ the frozen-per-step embeddings from a weak encoder whose receptive field is limited to small patches followed by global pooling. Let $h_\psi$ be a small regression network trained to predict $z$ from $s$. The training loss is
$$\mathcal{L} = \big\| \mathrm{W}(z_1 - h_\psi(s_1)) - \mathrm{W}(z_2 - h_\psi(s_2)) \big\|^2 ,$$
where $\mathrm{W}(\cdot)$ is batch or channel whitening of the residual. Because the residual is whitened, it always has unit variance. If the encoder routes shortcut information into $z$, the regressor explains it away. Whitening then amplifies whatever is left, which is mostly view-specific noise, and the alignment term between the two views gets worse. The only way to lower the loss is to supply invariant content that local statistics cannot predict.
Research Program
The work has three parts.
Part one: show the failure. We train whitening-based methods on Colorful-Moving-MNIST, RandBit and MultiDigits. On each we measure linear-probe accuracy for the semantic target, the effective rank of the features, and the eigenspectrum slope $\alpha$, defined by fitting $\lambda_i \propto i^{-\alpha}$ to the sorted covariance eigenvalues $\lambda_i$. We also measure how much of the feature variance a probe trained only on the shortcut can recover. If the whitened features are full rank and the shortcut probe recovers most of their variance, the core claim is supported.
This test also answers a question left open by $\alpha$-ReQ and Investigating Power laws in Deep Representation Learning. Those papers use $\alpha \approx 1$ as a label-free marker of good features, and α-ReQ itself notes that $\alpha \approx 1$ is necessary but not sufficient. We test whether shortcut-filled representations also sit in the $\alpha \approx 1$ band. If they do, that is a concrete and practically important case where the label-free metric is fooled.
Part two: build and characterize residual whitening. The design question that matters most is which weak encoder to use. We argue for restricting its receptive field rather than limiting its training time or depth. The shortcuts catalogued by Steering Self-Supervised Feature Learning Beyond Local Pixel Statistics, by Doersch-style chromatic aberration analyses Doersch et al., and by work on resampling fingerprints Noroozi et al. are all computable from small local windows. Objects are not. We sweep the window size from 3 to 32 pixels and the capacity of the regressor $h_\psi$, which ranges from linear to a 2-layer MLP.
The linear version is the label-free analogue of Controllable Feature Whitening. That method removes linear correlation between target and bias features but needs bias labels to build its bias encoder. Our version replaces the labeled bias encoder with an architecturally constrained one. The nonlinear version is needed precisely because of the Walsh argument above: removing only linear correlation with $s$ leaves nonlinear functions of the shortcut available to the encoder. ETF-Debias uses fixed geometric structures to quarantine known bias attributes. It shares our goal but, like CFW, requires annotations and operates in supervised training.
Part three: transfer to natural images. We move to ImageNet-100 and ImageNet with deliberately weakened augmentation, specifically without color distortion. There, Addressing Feature Suppression in Unsupervised Visual Representations reports that SimCLR loses 22.2% accuracy. We then check whether any gain persists under the standard augmentation recipe.
The trivial-solution literature enters as a lens, not a separate thread. An autoencoder that learns the identity map, the failure studied by Vincent et al. (2010) and addressed by Split-Brain Autoencoders and IRMAE, is the extreme case of solving a task with information that is available locally. Split-Brain blocks the copy path by changing what the target is. We do the analogous thing in feature space. We leave the task alone and remove from the target whatever a locally restricted model can produce.
We deliberately do not adopt IRMAE-style rank minimization. Low rank is exactly the dimensional collapse that On Feature Decorrelation in Self-Supervised Learning shows harms transfer. Whitening is kept for what it does well, which is keeping dimensions active.
Baselines
Naive reference. The do-nothing reference is SimCLR with standard augmentations. On Colorful-Moving-MNIST it reaches only about 15% digit accuracy according to PrCL. On RandBit, Intriguing Properties of Contrastive Losses reports that 10 shared bits drive MNIST linear evaluation from about 90% to below 20%. These numbers set the floor.
Plain whitening. The second baseline is plain whitening in its strongest available forms:
- W-MSE from Whitening for Self-Supervised Representation Learning, which reports 79.02% top-1 on ImageNet-100 with ResNet-18 and four positive views;
- CW-RGP from An Investigation into Whitening Loss, which reports 71.0% on ImageNet at 200 epochs and batch size 512;
- INTL from Modulate Your Spectrum, run through its official code, which reports 74.3% at 800 epochs.
This is the most direct competitor, because our method is plain whitening plus one module. For a fair comparison, our method uses the same whitening operator as each baseline, the same backbone and epochs, and a matched tuning budget. To isolate the effect of conditioning, we report results with each whitening variant as the base. On the shortcut benchmarks we have found no reported result for whitening methods on Lacuna, so we will establish those numbers ourselves.
Strongest anti-suppression method. This is PrCL. It adds a masked-patch inpainting branch to contrastive learning and reports 88.3% on Colorful-Moving-MNIST. It also reports only a 5.1% drop on ImageNet when color distortion is removed. PrCL is the method to beat on shortcut robustness. We will report both accuracy and training wall-clock and memory, since cost is our main argument for the alternative. Beating PrCL means matching its accuracy within one point at substantially lower cost, or exceeding it at comparable cost.
Adversarial lens. We also include the adversarial lens of Automatic Shortcut Removal. It improves Rotation and Jigsaw pretext tasks but is reported not to help SimCLR on ImageNet.
Ablations. Three ablations isolate the key ingredient:
- residual whitening with a linear regressor, which is the label-free CFW analogue;
- a weak encoder that has the full receptive field but is limited in depth or training time;
- a randomly initialized frozen weak encoder.
If the random encoder does as well as the trained local one, the gain comes from something other than modeling shortcuts.
Why not just…?
Why not just use plain whitening, since it already keeps every dimension active? The Walsh-basis argument says shortcut-only solutions are exactly whitened and exactly invariant, so they are optimal for W-MSE-style losses. The first experiment tests this directly. If plain W-MSE or INTL reaches PrCL-level accuracy on Colorful-Moving-MNIST and RandBit, the core claim is wrong, and the project becomes a positive result in favor of whitening that the file does not currently contain. We would report it as such.
Why not just add inpainting like PrCL? It works, and Intriguing Properties of Contrastive Losses explains why: reconstruction-based models must encode the full signal, so they resist shortcut bits. The objection is cost, not efficacy. A dense decoder runs on every image, whereas our weak encoder is a small local network plus an MLP regressor. If PrCL's overhead turns out to be modest at our scales, or our method falls well short of 88.3%, the case for our method becomes weak. The compute-matched comparison is designed to show which way this goes.
Why not just strengthen augmentations? Augmentations remove only the shortcuts someone anticipated. Colorful-Moving-MNIST defeats SimCLR's standard augmentation recipe at about 15%, and RandBit-style bits cannot be removed by any image augmentation. On natural images, however, strong augmentation is a real threat. Under the full recipe our gain may vanish. The ImageNet-100 runs with full augmentation will measure whether it does, and we will report a null result there honestly.
Why not just cut the projector head? Guillotine Regularization recovers information that the projector discards, but it recovers only what the trunk still holds. If the trunk itself learned the shortcut, cutting the head recovers nothing. We probe both trunk and projector layers for every method to separate these two failures.
Proposed Experiments
Synthetic benchmarks. On Colorful-Moving-MNIST, RandBit (sweeping $k$ from 2 to 16 bits and embedding dimension from 64 to 2048) and MultiDigits, we train SimCLR, W-MSE, CW-RGP, INTL, PrCL, the adversarial lens, and residual whitening with matched encoders and epochs, using three seeds each. The metrics are:
- linear-probe accuracy on the semantic target, meaning the digit, or the small digit in MultiDigits;
- shortcut decodability from the features;
- effective rank and $\alpha$;
- training cost.
Decisive check on RandBit. Plain whitening should fail whenever the embedding dimension is at most $2^k-1$. The failure may soften once the embedding dimension greatly exceeds the number of uncorrelated functions the shortcut can supply, because the encoder must then find other content. Residual whitening should be flat across $k$. Observing that crossover would confirm the capacity mechanism, not just a benchmark win.
Natural images. On ImageNet-100 with ResNet-18, we compare all methods with and without color distortion. Success means that residual whitening's accuracy drop from removing color distortion is no larger than PrCL's (PrCL reports 5.1% on ImageNet), and that clean accuracy stays within one point of W-MSE's 79.02%. If the method passes, we run a 200-epoch ImageNet comparison against CW-RGP's reported 71.0%, plus COCO detection transfer against CW-RGP's reported 40.7 AP.
Failure-mode probe. As a final check, we probe the weak encoder for the semantic labels. If a 16-pixel window already predicts digit identity well, the regressor will subtract real signal. The receptive-field sweep should then show accuracy falling as the window grows. That curve marks the regime where the method is safe to use.
References
- Intriguing Properties of Contrastive Losses
- $\alpha$-ReQ : Assessing Representation Quality in Self-Supervised Learning by measuring eigenspectrum decay
- Investigating Power laws in Deep Representation Learning
- Steering Self-Supervised Feature Learning Beyond Local Pixel Statistics
- Unsupervised Visual Representation Learning by Context Prediction
- Representation Learning by Learning to Count
- Controllable Feature Whitening for Hyperparameter-Free Bias Mitigation
- Navigate Beyond Shortcuts: Debiased Learning through the Lens of Neural Collapse
- Addressing Feature Suppression in Unsupervised Visual Representations
- Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion
- Split-Brain Autoencoders: Unsupervised Learning by Cross-Channel Prediction
- Implicit Rank-Minimizing Autoencoder
- On Feature Decorrelation in Self-Supervised Learning
- Whitening for Self-Supervised Representation Learning
- An Investigation into Whitening Loss for Self-supervised Learning
- Modulate Your Spectrum in Self-Supervised Learning
- Automatic Shortcut Removal for Self-Supervised Representation Learning
- Guillotine Regularization: Why removing layers is needed to improve generalization in Self-Supervised Learning