Same Hypothesis Class, Different Path: Depth-Reparameterized Probes That Find Minor Features

A linear probe trains one linear classifier on top of a frozen pre-trained network's features. It is the standard way to ask what a representation contains, and it has a known failure: it grabs the easiest, strongest signal. On data where an easy shortcut (a background, a pasted digit) co-occurs with the real object, the probe learns the shortcut. Fine-tuning that starts from such a probe then stays stuck on the shortcut. This proposal treats that failure as a question about the path optimization takes, not about which classifiers are possible. We keep the classifier strictly linear, so the set of functions the probe can express never changes. We change only how the linear weights are parameterized and optimized: the weight vector is written as the elementwise product of two vectors, and it is trained with sharpness-aware minimization (SAM, an optimizer that takes each step from the worst nearby point in weight space). A recent theory result predicts that this pairing emphasizes weak, "minor" feature directions early in training and returns to strong, "major" ones later. We read that early phase off the training trajectory itself and stop there.

The broader point is that path-dependent generalization, the idea that the route an optimizer takes decides how well the final model generalizes, has so far been studied almost only for networks trained end to end from scratch. A linear probe is the cleanest place to test it. The loss is convex, the endpoint of ordinary training is fully determined, and any change in out-of-distribution behavior at a fixed function class can come only from the path. If the effect is real, it yields a cheap, group-label-free probe for shortcut-laden data. It also yields a controlled experiment on whether path effects matter when the hypothesis class is held fixed.

Core Claim

Minor First, Major Last supplies the mechanism and the constraint. It proves that for single-layer linear models, gradient descent and SAM follow the same path to the $\ell_2$ max-margin classifier. So an ordinary linear probe cannot show any SAM-specific path effect. With depth two or more, in diagonal linear networks, $\ell_2$-SAM instead amplifies minor features early in finite time before major features take over. A Closer Look at Model Adaptation using Feature Distortion and Simplicity Bias supplies the motivation. It shows that vanilla probes latch onto simple shortcuts and that this traps later fine-tuning. That paper fixes the problem with added perturbations or weight averaging, and it treats the probe only as a finished endpoint.

Our claim has three parts. First, on frozen foundation-model features, shortcut signals concentrate in high-variance ("major") directions of the feature covariance. This is a measurable premise, not an assumption. Second, a depth-2 diagonal probe written in that covariance eigenbasis and trained with SAM passes through a window in which core, low-variance directions carry a larger share of the weight. Third, the location of that window can be detected from the probe's own weight trajectory, without group labels, and stopping there improves worst-group and randomized out-of-distribution accuracy over the probe's own converged endpoint.

Research Program

Let $Z$ be the frozen features, with covariance $\Sigma_Z$, and let $P$ hold the eigenvectors of $\Sigma_Z$ ordered by decreasing eigenvalue. The features are left unwhitened, so directions keep their natural scale. The probe is

$$f(z) = (u \odot v)^\top P^\top z, \qquad \min_{u,v}\ \max_{\|\epsilon\|\le\rho} L(u+\epsilon_u,\, v+\epsilon_v),$$

where $u, v$ are per-direction factors, $\odot$ is the elementwise product, $L$ is the logistic (or softmax) loss, and $\rho$ is the SAM perturbation radius. The effective weight on eigendirection $k$ is $w_k = u_k v_k$. The function class is exactly that of a linear probe. For multiclass problems, $u$ becomes a matrix with one column per class and $v$ is shared across classes.

The label-free stopping signal is the minor-mass ratio $m(t) = \sum_{k>K}|w_k(t)| / \sum_k |w_k(t)|$. Here $K$ is the number of top eigendirections that explain a fixed fraction of the variance, and $t$ is the training step. We stop at the peak of $m(t)$, restricted to steps where training accuracy has passed a threshold. This uses only training data and weights. Probes are tiny, so logging the full path is cheap. The computational objection raised against trajectory measures in Learning Trajectories are Generalization Indicators and Mutual Information Free Topological Generalization Bounds via Stability does not apply here.

As a secondary diagnostic, we compute the loss path kernel along the probe path. This is the integrated inner product of per-example loss gradients from Generalization Bound of Gradient Flow through Training Trajectory and Data-dependent Kernel. For a linear head it reduces to feature inner products weighted by loss derivatives. We ask whether it separates shortcut-dominated paths from core-dominated ones better than final accuracy does. This diagnostic addresses the static-readout problem raised by Probing the Probing Paradigm, where probes reached up to about 80% accuracy on task-irrelevant properties. It is supporting evidence only; the probe method is the main contribution.

On the theory side, the Moon et al. result assumes coordinate-aligned features. We will characterize when it carries over to the eigenbasis of correlated real features, and where it breaks.

Baselines

The naive reference point is a standard logistic linear probe solved to convergence with L-BFGS on the same frozen features. Its path is irrelevant by construction, so it anchors "no path effect." We also run the probe early-stopped under plain gradient descent at the step of best training-only criterion. This checks whether any early stopping helps. We expect plain gradient descent to fit major directions first, which would make its early stopping hurt.

The strongest existing competitor is the hardness-promoting probe family of Trivedi et al.: LP(VAT), LP(UDP), and LP(Soup). It targets exactly the same failure, needs no group labels, and is also a linear head. The strongest number in the file is that hp-LP+FT variants improved randomized out-of-distribution accuracy on Dominoes by "over 20%" relative to standard fine-tuning. We compare in two ways, probe-only and as initialization for LP+FT. Features, fine-tuning schedule, and hyperparameter search budget are identical across methods. Each method gets the same number of trials, selected by in-distribution validation only. Beating this baseline means higher worst-group or randomized out-of-distribution accuracy at no more than one point of in-distribution loss, averaged over seeds.

For Waterbirds, the file reports the multimodal contrastive fine-tuning of Mitigating Spurious Correlations in Multi-modal Models during Fine-tuning at "23% higher" worst-group accuracy than ERM with ResNet-50 and "32% higher" with ViT. That method fine-tunes the encoder and uses more compute and supervision, so it serves as an upper reference, not a like-for-like competitor. No reported result was found on Lacuna for vanilla linear-probe worst-group accuracy on Waterbirds, so we establish that number ourselves.

In the few-shot CLIP regime, where we only check for no regression, the baseline is LP++. The file gives its roughly 2-second 16-shot ImageNet training time but no per-dataset accuracies ("no reported number found").

The ablations isolate the key ingredient. The first is a single-layer probe trained with SAM, which theory says must match gradient descent and so serves as a sanity check. The second is the diagonal depth-2 probe trained with plain gradient descent, which adds depth without SAM. The third is the full method with oracle stopping chosen on the out-of-distribution split, which separates the value of the path from the quality of the stopping rule. The fourth is the full method in a random orthogonal basis instead of the eigenbasis, which tests whether the basis carries the effect.

Why not just…?

Why not just whiten the features and train an ordinary probe? Whitening puts every direction on equal scale, so the max-margin solution no longer favors high-variance directions. It is the most serious threat, because it may capture most of any gain with no trajectory at all. We include whitened L-BFGS and whitened-plus-$\ell_2$ probes as full baselines. If they match our worst-group accuracy, we will report that the effect is a scaling effect, not a path effect, and drop the path claim. One reason to expect a difference is that whitening also amplifies noise directions uniformly, whereas the SAM path is predicted to amplify selectively. The random-basis and oracle-stop ablations test that distinction.

Why not just use an $\ell_1$-regularized probe, since diagonal parameterizations are known to favor sparsity? Sparsity in the eigenbasis could select a few strong directions, which is the opposite of what we want, or a few weak ones. That outcome is empirical, so we run $\ell_1$ probes in the same eigenbasis over a matched regularization sweep. The diagonal gradient-descent ablation sits between the two and shows whether SAM adds anything beyond the diagonal parameterization.

Why not just use hardness-promoting probes, which already work? They do work on Dominoes by the file's numbers. Their guidance, however, comes from perturbations and averaging, and they offer no signal about when a probe is on a shortcut. If they match or beat us on accuracy, the remaining contribution is the controlled test of path dependence and the label-free diagnostic. We would state that plainly rather than claim a better probe.

Why not just use held-out validation to choose among probes? In-distribution validation accuracy hides subgroup failure, as The Effect of Model Size on Worst-Group Generalization documents on Waterbirds. In few-shot settings there is barely any validation set to use. We report how well validation-selected stopping and trajectory-selected stopping each track the oracle stopping point.

Proposed Experiments

We use frozen CLIP ResNet-50 and ViT encoders, plus ImageNet-pretrained ResNets, on four settings. Dominoes (CIFAR-10 objects paired with MNIST digit shortcuts) is scored by randomized out-of-distribution accuracy. Waterbirds and CelebA are scored by worst-group accuracy. The ImageNet baby–pacifier subset is scored by worst-case adjusted intersection-over-union. The 11-dataset CLIP few-shot suite, at 1 to 16 shots, checks that in-distribution accuracy does not drop. We use five seeds throughout.

The first decisive check tests the premise before any method comparison. We project the known spurious attribute onto the eigenbasis and ask whether it lives mostly in the top-$K$ directions. If it does not on a given dataset, the method should not help there, and a failure there is predicted rather than embarrassing.

The second check tests whether the SAM window exists. It is confirmed if the minor-mass ratio $m(t)$ rises and then falls under diagonal SAM, and not under single-layer SAM or diagonal gradient descent. We also require that worst-group accuracy along the path peak near the peak of $m(t)$.

The third check tests the label-free stopping rule. It succeeds if trajectory-stopped diagonal SAM beats the converged vanilla probe, the whitened probe, and hp-LP, both alone and as LP+FT initialization. Its worst-group or randomized out-of-distribution accuracy should also come within a small margin of oracle stopping.

The fourth check compares the loss-path-kernel statistic of each run with final probe accuracy. The question is which better ranks runs by out-of-distribution accuracy, measured as a rank correlation across all runs.

The method fails if whitening matches it, or if the window appears only under oracle stopping. The failure would still be informative: it would show that, at fixed linear capacity, the optimizer's path adds nothing beyond feature scaling.