Linear Probes Inherit an Arbitrary Gauge: Scale Symmetry as a Hidden Source of Shortcut Reliance

A linear probe is a single linear classifier trained on top of a frozen pretrained network. It is the standard way to ask what a representation "knows." But networks built from ReLU units, which pass positive inputs and zero out negative ones, have a hidden freedom. You can multiply any hidden channel by a positive constant and divide the weights reading from it by the same constant, and the network computes exactly the same function. The probe does not see it that way. Its regularizer, and the implicit bias of gradient descent toward large-margin solutions, both measure weight size in the coordinates it is handed. A channel that happens to be large is cheap to use, and a channel that happens to be small is expensive. We propose that this arbitrary scale accounts for a measurable share of the shortcut reliance attributed to pretrained representations: the probe's preference for easy, spurious features such as image backgrounds over the actual object. If so, two functionally identical backbones can receive different verdicts from the same probing protocol.

The claim matters because probing results are routinely read as facts about representations, and simplicity bias in probes is known to propagate downstream. A Closer Look at Model Adaptation using Feature Distortion and Simplicity Bias shows that a probe that settles on a shortcut locks subsequent fine-tuning into it. We will measure how much of that preference is set by function and how much by the backbone's scale convention. We then test one remedy that follows directly from the analysis: averaging probes across many equally valid rescalings of the backbone.

Core Claim

On the Symmetries of Deep Learning Models and their Internal Representations defines the intertwiner group $G_{\text{ReLU}}$, the set of linear maps that pass through a ReLU unchanged: channel permutations and positive per-channel scalings. It uses this group to build similarity measures that ignore these redundancies. That work supplies our method. It studies only in-distribution accuracy and interpretability, and never asks what the symmetry does to a probe trained on top. The key observation is a one-line equivalence. Rescaling the frozen features $z$ by a positive diagonal matrix $D$ and training an $L_2$-regularized probe $w$ on $Dz$ is the same as training on the original $z$ with a reweighted penalty:

$$\min_{w}\; \mathcal{L}(w^\top D z) + \lambda \|w\|^2 \;\;\Longleftrightarrow\;\; \min_{v}\; \mathcal{L}(v^\top z) + \lambda \|D^{-1} v\|^2, \qquad v = Dw.$$

Here $\mathcal{L}$ is the classification loss and $\lambda$ the penalty strength. The same reweighting governs the unregularized max-margin limit. Permutations leave the probe unchanged, so the scaling part of $G_{\text{ReLU}}$ is the only part that matters. The backbone's training process, including weight decay (which the symmetry paper notes partly pins down scale), therefore silently chooses the probe's regularizer.

Our claim has three parts. First, across the $G_{\text{ReLU}}$ orbit of a fixed backbone, which leaves pretrained behavior exactly unchanged, probe worst-group accuracy on shortcut benchmarks varies by an amount comparable to the gaps that separate published mitigation methods. Second, the realized scale of shortcut-carrying channels relative to core channels predicts which feature a probe adopts across seeds and architectures. Third, averaging probes trained under random orbit rescalings reduces shortcut reliance at matched cost. The intuition is that each rescaling makes a different subset of features cheap, so the average draws on more features than any single probe.

Research Program

The first component is a controlled demonstration. We take an ImageNet-trained ResNet-50, rescale the channels of its final residual block by positive constants, and compensate in the original classification head. We then verify that ImageNet logits are unchanged to numerical precision while a downstream probe's worst-group accuracy moves. This makes the claim concrete: same function, different probing verdict.

The second component maps the orbit. We sample diagonal rescalings with log-normal entries of increasing spread. We also construct an oracle rescaling that uses group labels to inflate or shrink channels correlated with the spurious attribute. Across these, we record the range of probe outcomes. We also locate common preprocessing within that range. Per-feature standardization (z-scoring) picks one specific gauge. Full whitening goes beyond $G_{\text{ReLU}}$ to arbitrary invertible maps, which mirrors the over-flexibility that Evaluating Representational Similarity Measures from the Lens of Functional Correspondence documents for linear predictivity as a similarity measure.

The third component is predictive. Across independently trained backbones (CIFAR-scale ResNets trained from scratch with varied seeds and weight decay, plus available ImageNet ResNets), we compute a per-channel "shortcut alignment" score and the channel's realized scale. We then test whether their scale-weighted combination predicts probe shortcut reliance better than three alternatives: standard CKA, Procrustes, or $G_{\text{ReLU}}$-CKA similarity to a reference model.

Data symmetries generate drifting similarity matrices in manifold-tiling neural codes found that sparsity-inducing training ($L_1$ penalties, noisy SGD) amplifies gauge-driven drift in similarity structure. We therefore also test whether sparser backbones show larger probe gauge sensitivity. This would extend their low-dimensional synthetic finding to real downstream readouts.

The fourth component is the remedy, a gauge-marginalized probe. We train $K$ probes, each on features rescaled by an independent random element of $G_{\text{ReLU}}$, map each back to original coordinates, and average their logits. The rescaling spread is the single hyperparameter, selected on a validation set without group labels.

Baselines

The naive reference is a standard logistic-regression probe on raw penultimate features with $L_2$ penalty tuned on in-distribution validation accuracy. This is the protocol most probing studies use.

The second baseline is the same probe on z-scored features. It is the most dangerous competitor, because it already fixes the scale gauge by convention, and it costs nothing. A third is the whitened-feature probe.

The strongest existing probe-level method is hardness-promoting linear probing from A Closer Look at Model Adaptation using Feature Distortion and Simplicity Bias. It comes in three variants: virtual adversarial perturbations of embeddings, uncertainty-driven perturbations, and weight soups of sparse probes. LP(Soup) is the closest analogue to our ensemble, since it also averages diverse probes. The only number the file reports for hp-LP is a gain of over 20% in Randomized-OOD accuracy on Dominoes for hp-LP followed by fine-tuning, relative to standard fine-tuning. No reported result was found on Lacuna for hp-LP probes alone without fine-tuning, so we will reproduce them.

We also run MixPro from Eliminating Spurious Correlations from Pre-trained Models via Data Mixing, which trains a probe on interpolated embeddings. The file reports only an average improvement of 3.9% to 4.3% over projection baselines in 2-shot and 4-shot Waterbirds, and no reported result was found on Lacuna for its absolute worst-group accuracy. Finally, we run Spectral Decoupling from Gradient Starvation: A Learning Proclivity in Neural Networks applied to probe logits.

Fairness requires the same frozen features, the same training and validation splits, the same total number of probe-training runs (our $K$ probes against $K$ soup members or $K$ tuning trials), and the same label access. No method may use group labels for model selection unless all do. We report both regimes.

Two ablations isolate the key ingredient. A seed ensemble of $K$ probes differing only in initialization and data order tests whether gauge, rather than ensembling per se, supplies the diversity. A random-rotation ensemble should produce identical probes under $L_2$ and so serves as a sanity check. Rotations are also not realizable by the same ReLU architecture, which separates "symmetry of the network" from "arbitrary linear map."

As full-training reference points, not same-budget competitors, we cite the following worst-group results: on Waterbirds-CC, ERM at 7.0%, Just Train Twice at 26.5%, and DivDis at 82.4% (Diversify and Disambiguate: Learning From Underspecified Data); on CelebA, Spectral Decoupling at 83.24% (Gradient Starvation) and Alignment-Gated Suppression at 93.95% (Regulating Internal Alignment Flows for Robust Learning Under Spurious Correlations). Beating the probe-level baselines means higher worst-group or Randomized-OOD accuracy than both z-scored probing and the best hp-LP variant, outside seed variance, at matched cost.

Why not just…?

Why not just z-score the features and declare the gauge fixed? This is a real threat. Z-scoring is cheap and removes dependence on the backbone's realized scale. The proposal treats it as a hypothesis rather than a refutation. If z-scored probes sit near the favorable end of the orbit on every benchmark, the diagnostic result still stands: unstandardized probing verdicts are gauge artifacts, and published comparisons that skip standardization should be rechecked. The ensemble remedy would then be unnecessary, and we would say so. If z-scored probes sit mid-orbit or at the unfavorable end, standardization is merely one arbitrary gauge choice. The orbit-mapping experiment decides this directly.

Why not just use a nonlinear MLP probe? The Pitfalls of Simplicity Bias in Neural Networks shows that ReLU networks trained by gradient descent become fully invariant to complex features when a simple one exists, so extra capacity does not remove the preference. The Illusion of Intent reaches only 0.721 AUROC with a two-layer contrastive probe on its paired adversarial set, and it does so by changing the training data, not just capacity. An MLP probe also stops being a test of linear decodability. We include a two-layer MLP probe as a secondary comparison on Dominoes.

Why not just use hardness-promoting probes, which already target probe simplicity bias? They are the closest competitor and may be complementary. Their perturbations are applied in Euclidean embedding space, which the equivalence above shows is itself gauge-dependent. The head-to-head comparison, and the combination of hp-LP run inside each gauge sample, will show whether gauge marginalization adds anything beyond them.

Why not just project out the spurious direction or reweight features with group labels? Projection discards dimensions that may carry target signal, which is how MixPro reports beating PRO2 and DFR. Label-driven reweighting is also, by our equivalence, exactly a choice of gauge. That makes our oracle-rescaling experiment an upper bound on what any label-free gauge choice could achieve, rather than a competitor to it.

Proposed Experiments

We use frozen ImageNet ResNet-50 and ResNet-18 backbones on Dominoes and the MNIST-CIFAR concatenation, measuring Randomized-OOD accuracy with the digit-label correlation broken. We also evaluate Waterbirds, Waterbirds-CC, and CelebA by worst-group accuracy. CIFAR-10 ResNets trained from scratch across seeds and weight-decay values support the predictive analysis.

A small synthetic check uses two linearly decodable features with known margins, one fully predictive and one 95% predictive, echoing the noisy-shortcut design of $\hat{\text{LMS}}$-k. It verifies that the probe switches features exactly when the scale ratio crosses the margin ratio implied by the max-margin solution. A secondary run applies the same analysis to the per-channel gains of the final LayerNorm in a Vision Transformer, where an analogous scale freedom exists.

The checks are decisive in both directions. If the full orbit of exact reparametrizations moves Waterbirds worst-group accuracy by only a point or two, the core claim fails. If realized channel scale does not predict probe shortcut use beyond what $G_{\text{ReLU}}$-CKA similarity already explains, the mechanistic story fails. If the gauge ensemble does not beat both the seed ensemble and z-scored probing at matched cost, the remedy fails, and the contribution reduces to an auditing result about probing practice. That result would still be a useful finding about how much of "the representation's shortcut" belongs to the representation at all.