Where a Frozen Probe Fails Under Domain Shift: Measuring the Joint Error of Linear Heads

When a pretrained encoder is frozen and only a linear classifier is trained on top of it (a linear probe), the probe's mistakes on a new, unlabeled domain can come from three different places. The hyperplane may simply sit in the wrong place for the new domain, even though a better one exists. Or no single hyperplane can serve both the old and new domains, because the frozen features place a target "6" where source "9"s live. Or the target classes are not linearly separable in the frozen features at all. These three cases call for different fixes: moving only the classifier, changing how source data is used, or changing the encoder itself. Practitioners currently cannot tell them apart without target labels, so they guess, often by running encoder-updating adaptation methods that damage pretrained features.

We propose to split target error into these three parts using domain adaptation theory's ideal joint error, the lowest combined source-plus-target error any single classifier in the allowed family can achieve. That quantity is normally unmeasurable for deep networks. It becomes a convex, exactly computable quantity once the encoder is frozen and the classifier family is linear. The program has three steps. First, measure the decomposition with oracle target labels across encoders and standard shift benchmarks. Second, build label-free estimators of each part. Third, test whether those estimates choose the right adaptation strategy better than estimating target accuracy alone. The core claim is that, for frozen foundation-model features, the joint error of linear heads is both a ceiling on head-only adaptation and a usable signal for deciding when the encoder must change.

Core Claim

Let $z = f(x)$ be frozen features. Let $w_S$ be the probe trained on labeled source data, $w_J$ the oracle probe trained on labeled source and target together, and $w_T$ the oracle probe trained on target labels only. Write $\epsilon_S, \epsilon_T$ for source and target error. Then the source-only probe's target error splits exactly into three non-negative-in-practice parts:

$$\epsilon_T(w_S) = \underbrace{[\epsilon_T(w_S) - \epsilon_T(w_J)]}_{\text{misplaced head}} + \underbrace{[\epsilon_T(w_J) - \epsilon_T(w_T)]}_{\text{entanglement}} + \underbrace{\epsilon_T(w_T)}_{\text{rigidity}}.$$

Each part matches a term in the three-term bound of Analysis of Representations for Domain Adaptation, which bounds target error by source error, a domain divergence, and the ideal joint error $\lambda = \min_h \epsilon_S(h)+\epsilon_T(h)$. Here $\lambda_{LP} = \epsilon_S(w_J)+\epsilon_T(w_J)$ is that joint error restricted to linear heads on frozen features. Two parts of the split, entanglement and rigidity, together bound what any head-only method can recover.

Our first hypothesis is that, across modern encoders, the three parts vary enough between (encoder, shift) pairs that no single adaptation strategy is uniformly right. Our second hypothesis is that label-free estimates of the parts predict which strategy helps.

Relation to Prior Work

The motivating evidence comes from deep domain adaptation. How does the Combined Risk Affect the Performance of Unsupervised Domain Adaptation Approaches? shows that joint error rises during adversarial alignment and controls it with a mixup-based proxy classifier (E-MixNet). Unsupervised Domain Adaptation via Minimized Joint Error restricts the search to classifiers that fit both source data and pseudo-labeled target data (MJE). Domain Adaptation and Entanglement: an Optimal Transport Perspective explains the rise as label entanglement caused by forcing marginal distributions together.

All three papers backpropagate through the encoder. As a result, joint error is a moving target and cannot be measured cleanly. We freeze the encoder precisely so that $\lambda_{LP}$ becomes a fixed, computable property of the encoder and the domain pair. We do not propose another joint-error-minimizing training method.

The constraining evidence comes from Practicality of generalization guarantees for unsupervised domain adaptation with neural networks, which reports two things: the additive bound with a joint error term is the tightest, yet that term cannot be evaluated without labels, and bound minima do not track the best checkpoints. This rules out using a bound value for model selection. We ask a different question: whether the composition of the error predicts which remedy works.

The closest structural ancestor is A PAC-Bayesian Approach for Domain Adaptation with Specialization to Linear Classifiers. It bounds the target risk of a distribution over linear classifiers using a disagreement measure and a joint term $\lambda_\rho$, on raw bag-of-words and toy inputs. We reuse its disagreement measure as one estimator, now applied to deep frozen embeddings.

On the probing side, Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution shows that fine-tuning distorts features: it was 7% worse than linear probing out of distribution, and LP-FT was about 10% better than fine-tuning out of distribution on average across 10 shifts. That paper argues for freezing, but it uses labeled target data and never asks when freezing hits a ceiling. Our rigidity term is exactly that ceiling.

On Pitfalls of Test-Time Adaptation identifies a "classifier bottleneck", where adapting features fails when the head is misaligned with the target domain. Our misplaced-head term gives that bottleneck a measurable size.

LogME and LiDAR predict probe quality without training probes. LogME needs target labels, and LiDAR works within a single domain. We borrow LiDAR's label-free spirit for the rigidity estimate only.

Research Program

Oracle map. We compute the exact decomposition for frozen CLIP, DINO and MoCo-v2 encoders (ResNet and ViT where available) on Office-31, Office-Home, VisDA-2017 and DomainNet. We use L2-regularized multinomial logistic probes with matched regularization across $w_S$, $w_J$ and $w_T$, on held-out splits, so that the oracle terms are not inflated by overfitting. This produces the first measured values of $\lambda_{LP}$ for foundation-model features. It also tests whether entanglement is common in practice or mostly a digit-rotation curiosity.

Label-free estimators.

  • Misplaced head. We estimate this from how far apart near-optimal source probes fall on target. We sample probes within a small slack of the source optimum, in the PAC-Bayesian sense, and measure their target disagreement. High disagreement means the target data sit where source data leave the head underdetermined.
  • Rigidity. We estimate this from target-only cluster structure: how well a linear classifier fits clusters found in target features, and how cleanly separated those clusters are.
  • Entanglement. This is the delicate one. If target pseudo-labels came from $w_S$, a joint probe would fit them trivially, so the estimate would be circular. Instead, we obtain target pseudo-labels from target clusters matched to source class prototypes by optimal transport, which does not depend on the source hyperplane. We then measure the residual error of a single linear probe fit jointly to source labels and these pseudo-labels.

Remedy selection. Each estimated profile maps to one of three actions. A large misplaced-head term calls for head-only adaptation on frozen features, such as an MCD- or MJE-style head. A large entanglement term calls for class-conditional reweighting or dropping conflicting source classes. A large rigidity term calls for encoder-updating adaptation.

Baselines

The naive reference point is the source-only linear probe $w_S$ with no adaptation. Every decomposition and selection result is reported relative to it.

The strongest competitor for the remedy-selection question is a target-accuracy estimator built from agreement between independently trained source probes, scaled the way Why do classifier accuracies show linear trends under distribution shift? relates in- and out-of-distribution accuracy. It is the obvious label-free signal practitioners would reach for. It is given the same features, the same probe ensemble, and the same selection rule form: a threshold tuned on a held-out set of (encoder, shift) pairs disjoint from the test pairs.

For the adaptation arms, the strongest existing end-to-end methods are MJE, which reports 81.6% average accuracy on VisDA-2017 Unsupervised Domain Adaptation via Minimized Joint Error, and E-MixNet, which reports 70.6% on Office-Home and 90.2% on Office-31 How does the Combined Risk Affect the Performance of Unsupervised Domain Adaptation Approaches?. These numbers come from those papers' own backbones. No reported result found on Lacuna for head-only adaptation on frozen CLIP or DINO features on these benchmarks, so we rerun all arms ourselves with identical backbones, data splits, and a fixed tuning budget per arm. Tuning is unsupervised: we use the same small number of reverse-validation trials for every method, never target labels, because On Pitfalls of Test-Time Adaptation shows that label-informed tuning hides drops of over 60%. LP-FT on source labels is included as the distortion-aware supervised alternative.

Two ablations isolate the key ingredients. The first replaces the three-part profile with a single estimated-accuracy score, which tests whether the decomposition, rather than error size, carries the signal. The second derives entanglement pseudo-labels from $w_S$ instead of OT-matched clusters, which tests whether removing circularity matters.

Beating the strongest baseline means two things. The decomposition-driven selector must achieve lower regret than the agreement-based selector, where regret is the target accuracy lost relative to the oracle-best of the three remedies, averaged over held-out (encoder, shift) pairs with paired bootstrap intervals. The selector must also beat every fixed "always use X" policy.

Why not just…?

Why not just estimate target accuracy and adapt when it is low? A low accuracy estimate says the probe is failing, but not why. A misplaced head and a rigid encoder can produce the same accuracy and call for opposite actions. We do not assume this distinction matters; the first ablation tests it directly. If a single accuracy score selects remedies as well as the profile does, our claim fails and we will report that.

Why not just always run end-to-end joint-error adaptation like MJE? Encoder updates carry the distortion cost documented in Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution. On pairs where the head is merely misplaced, we expect head-only adaptation to match or beat MJE at a fraction of the compute. Conversely, we expect MJE to win where rigidity dominates. The oracle map shows how often each case occurs. If MJE wins nearly everywhere, the selector adds nothing beyond cost savings, and we will say so.

Why not just always self-train the probe on frozen features? This is a real threat. Self-training is cheap and may recover most of the misplaced-head error. If it consistently reaches the joint-probe error $\epsilon_T(w_J)$, the decision collapses to "self-train, then check rigidity," and only the rigidity estimator matters. The oracle comparison of self-trained probes against $\epsilon_T(w_J)$ decides this. We would then narrow the contribution accordingly.

Decisive Checks

The program rests on three checks. First, the oracle map must show substantial variation in which part dominates across (encoder, shift) pairs. If misplaced-head error dominates everywhere, the selection question is moot. Second, each label-free estimator must correlate with its oracle counterpart, measured by Spearman rank correlation over held-out pairs. The entanglement estimator must also beat its circular ablation. Third, the decomposition-driven selector must achieve lower regret than the agreement-based accuracy selector and than every fixed policy. A final sanity case uses rotated digits, where entanglement is constructed by design: there, the entanglement estimate must rise with rotation angle while the misplaced-head estimate does not.