When a model must keep learning new classes while the world's appearance keeps changing, two problems meet. The first is crowding: new classes land on top of old ones in the model's feature space. The second is style entanglement: features record how an image looks (lighting, weather, medium, sensor noise) alongside what it shows. This note argues that on frozen pretrained backbones, much of the apparent crowding is style. A class learned from sketches sits near other sketch classes, and a foggy test image of an old class drifts toward whatever classes were learned in similar conditions. If that is true, we do not need to carve empty room into a frozen feature space, which is impossible without retraining. We can change how distances are measured, so that differences along style directions count for less. We can also estimate those directions online from the unlabeled test stream.
Concretely, we propose a backpropagation-free classifier on a frozen backbone. It keeps one Gaussian per class, meaning a mean plus a covariance shared across classes. The key change is that the shared covariance is inflated along a "nuisance subspace": the directions in which features move when an image is perturbed without changing its content. That subspace is re-estimated from paired perturbations of incoming test images. Test-time adaptation is allowed to shift features only inside it. Because the method never moves features along content directions, it cannot pull class boundaries onto wrong clusters, which is the failure that hurts entropy-based adaptation. We also construct the benchmark this question needs: classes arrive in sessions, each possibly in its own style, while the test stream drifts through corruptions and domains.
Core Claim
The claim has three parts, and each can be tested. First, on frozen backbones under drift, a measurable share of the confusion between classes from different sessions comes from mean differences that lie in a low-dimensional nuisance subspace. That subspace can be found from content-preserving perturbations, without domain labels. Second, soft down-weighting of that subspace inside a Gaussian classifier removes this share of the confusion. Deleting the subspace outright does not; it reproduces the trade-off reported for hard style suppression. Third, where the nuisance directions lie depends on where the target data sits in feature space. A subspace estimated online from the test stream should therefore beat one estimated from source-side augmentation.
The mechanism fits in one equation. Let $z$ be the frozen backbone's feature for an image, $\mu_c$ the stored mean of class $c$, $\Sigma$ the shared within-class covariance accumulated over training sessions, $U$ an orthonormal basis for the $k$-dimensional nuisance subspace, $\lambda \ge 0$ an inflation strength, and $b_e$ an offset for the environment $e$ that the sample is routed to, constrained to lie in the span of $U$. The class score is
$$ s_c(z) = -\,(z - b_e - \mu_c)^\top \left(\Sigma + \lambda U U^\top\right)^{-1} (z - b_e - \mu_c). $$
The Woodbury identity shows that inflation reduces the weight on displacements inside $U$. For large $\lambda$, that weight approaches zero, which is hard projection, but the information is kept rather than discarded. $U$ is formed from the top-$k$ left singular vectors of a matrix of feature differences $f(x') - f(x)$, where $x'$ is a mild content-preserving perturbation of $x$ (brightness, blur, noise, color jitter). The singular vectors are updated as an exponential moving average over the stream.
Positioning Against Prior Work
The closest work is D$^2$O. It identifies a nuisance-sensitive subspace in CLIP features from finite differences of paired perturbations. It softly suppresses that subspace for cache retrieval, and it tracks per-environment logit biases, all without training. We reuse its subspace estimator; it supplies our method. D$^2$O also reports that style-suppressed features help nearest-neighbor retrieval but that raw features are better for Gaussian modeling. That finding constrains us, and it is why we inflate the covariance rather than project features. Our design is a direct test of whether the two findings can be reconciled. D$^2$O works on a fixed label set and never meets new classes, so the style-induced crowding we target falls outside its scope.
CLIPCEIL also separates channels that vary with domain from channels that vary with class. It trains an adapter that shrinks inter-domain variance and grows inter-class variance. It needs several labeled source domains, trains offline, and uses a fixed class set. We require none of these, at the cost of a subspace estimate that is heuristic rather than supervised.
On the crowding side, three works set up the problem. Prospective Representation Learning squeezes base classes during base training to leave room for future classes. That requires retraining the backbone, and on frozen foundation models this route is closed. TAILS keeps the backbone frozen. It trains a small module that uses stored task anchors to correct features and reduce cross-task confusion. It is trained offline with labels and evaluated without streaming drift. Advancing Analytic Class-Incremental Learning shows that closed-form classifiers on frozen features avoid forgetting; our classifier belongs to this family. None of the three models the test stream changing style.
On the drift side, CoTTA, SoTa-DiT, and OoPk adapt to non-stationary streams but assume a closed class set. Tent documents that entropy minimization reinforces its own confident mistakes. That result motivates confining adaptation to nuisance directions. T3A recomputes classifier templates from confident test samples with no gradients. The research file notes that such head-only methods cannot fix overlap already present in the features. We accept that frozen features are fixed, but we change the metric on them, which is more than re-centering templates.
Representation rigidity enters as a constraint and a predicted failure mode, not as something we solve. If the frozen backbone lacks the content features, as reported for pathology and medical imaging in Few-shot Adaptation of Medical Vision-Language Models, reweighting distances cannot help. We expect and will report failure there.
Research Program
The work has three parts: a benchmark, a diagnostic, and the method.
The benchmark is "class-incremental under drift" (CI-drift), built only from datasets in the file, and it has three protocols. The first is CIFAR-100 split into 10 sessions of clean training classes, evaluated on a stream of CIFAR-100-C images of all classes seen so far. The corruption type cycles in the continual order used by CoTTA. The second is ImageNet-R split into 10 sessions of 20 classes, evaluated on a stream mixing renditions with ImageNet-C-style corruptions of the matching ImageNet classes. The third is the one that targets arrival bias. DomainNet's 345 classes are split into sessions, and each session's training data comes from a single domain, such as sketches for session 3. The test stream then draws every seen class from every domain. Here a class's stored mean carries the style of its arrival session. PUG serves as a controlled probe, because illumination, orientation, and texture can be varied one at a time.
The diagnostic runs before any method claims. For a frozen ViT-B/16 and a frozen CLIP ViT-B/16, we measure two quantities. The first is the fraction of the energy in the mean difference between each pair of classes from different sessions that lies inside $U$. The second is how cross-session confusion changes when displacements inside $U$ are discounted. If the fraction is small, for instance if discounting $U$ fixes under a tenth of cross-session errors, the core claim is false. We would then report that crowding on frozen backbones is genuinely about content. We also measure the principal angles between $U$ and the between-class scatter. Small angles mean the perturbations are erasing class information, which is the main way this method can fail.
The method is the classifier above, with three moving parts. Class means and the shared covariance update in closed form each session. $U$ and the environment offsets update online from the stream. Environments are found by clustering the coordinates $U^\top z$, as D$^2$O does. No backbone weights change and no gradients are computed.
Baselines
The naive reference is a nearest-class-mean classifier on frozen features with no test-time adaptation. It shows how much crowding and drift cost in the first place.
The strongest simple competitor is a frozen-feature Gaussian classifier with a shared covariance, in the analytic style of Advancing Analytic Class-Incremental Learning. It is strong because a shared covariance already down-weights any direction with high within-class variance, and style often has high variance. It will use the same backbone, sessions, and ridge regularization budget, with covariance shrinkage tuned on a held-out split of the first session. That split is also where our $\lambda$ and $k$ are tuned. Our method must beat it on final accuracy on CI-drift. The margin must exceed the spread across three seeds, and the gain must come from the cross-session confusion rate rather than within-session errors.
The strongest method from the literature is TAILS. We run it with its trained shaping module and then stack D$^2$O's training-free operator on top at test time. This combination is the obvious way to cover both crowding and style, so beating it is the real test. Both get identical frozen backbones and session data. TAILS gets its own training budget, and our method's stream-side computation, roughly $k$ extra forward passes per perturbed batch, is reported next to its cost. The file gives no reported result on Lacuna for TAILS, D$^2$O, or analytic CIL on any class-incremental benchmark with streaming drift, because the setting does not yet exist. The only nearby numbers are these. TAILS reports gains of "up to 5.05 percentage points" in final accuracy on static ImageNet-R/A, CARS-196, and VTAB-50. D$^2$O reports "58.28%" on PUG with ADAPT, against "51.87%" without it. CoTTA reports "32.5%" average error on closed-set continual CIFAR100-C. We will rerun all three in CI-drift rather than borrow these numbers.
For closed-set context we include CoTTA and T3A, each paired with the Gaussian classifier's per-session means. Four ablations isolate the key ingredient. The first estimates $U$ from source-side augmentations only, which tests the claim that nuisance directions are local to the target. The second replaces inflation with hard projection, which tests the D$^2$O tension. The third removes the environment offsets. The fourth uses a random $k$-dimensional subspace, which controls for plain regularization.
Why not just…?
Why not just use a shared-covariance Gaussian classifier and stop? This is the most serious threat. If within-class scatter from the training sessions already contains the relevant style directions, inflation adds nothing. The DomainNet protocol decides it. There, each class's training scatter covers only one style, so the covariance cannot anticipate the test styles. The source-augmentation ablation separates "more covariance" from "covariance estimated at the target." If our method gains nothing over this baseline across all three protocols, the proposal reduces to "use LDA," and we will say so.
Why not just augment each training session with style perturbations? Folding perturbed copies into $\Sigma$ is nearly the same as our source-side $U$. Our claim is specifically that target-side estimation matters, because where the nuisance directions lie depends on where the target sits in feature space. The source-versus-stream ablation measures this directly, including on corruptions such as Gaussian noise that the training augmentations did not cover.
Why not just stack D$^2$O on TAILS? D$^2$O corrects logits and retrieval inside a fixed label space. Its environment biases are tracked per class for classes that all exist from the start. Nothing in it stops a class learned in sketches from absorbing sketch-style test images of other classes. We run this stack as the main baseline. The per-session confusion analysis will show whether its errors concentrate on classes that arrived in the same style.
Why not just run CoTTA or Tent on the stream? These methods update weights using pseudo-labels, and the file documents error accumulation in both: TENT-continual reaches "20.7%" error versus "16.2%" for CoTTA on CIFAR10-C. CoTTA's cost of up to 32 augmentations per step is also prohibitive. Neither handles new classes. We report them as context, not as the target to beat.
Proposed Experiments and Decisive Checks
The primary metrics are average incremental accuracy and final accuracy on the three CI-drift protocols. We also report the cross-session confusion rate, broken down by whether the true and predicted classes shared an arrival style, and on PUG the accuracy for each nuisance factor. Three results would be decisive.
The first is the diagnostic. The share of cross-session confusion explained by $U$ should be substantial on DomainNet and small on clean in-distribution tests. If it is flat across the two, the "crowding is style" hypothesis fails.
The second is the tension check. Soft inflation should keep the Gaussian classifier's clean accuracy while hard projection lowers it. That would confirm D$^2$O's finding and show that inflation resolves it. If hard projection is equally good, the reconciliation claim is unnecessary.
The third is the locality check. A stream-estimated $U$ should beat a source-estimated $U$ most clearly on corruptions absent from the augmentation set. It should also have measurably larger principal angles to the source $U$ on those corruptions.
We will additionally report one medical rigidity case, adapting the setup to the scanner shifts in the file's medical benchmarks. There we predict little gain, because the backbone lacks the content features. That outcome would mark where metric reshaping stops and new feature learning becomes necessary.
References
- D$^2$O: A Dual Debiasing Operator for Training-Free Test-Time Adaptation of Vision–Language Models
- CLIPCEIL: Domain Generalization through CLIP via Channel rEfinement and Image-text aLignment
- Prospective Representation Learning for Non-Exemplar Class-Incremental Learning
- Task-Anchored Representation Shaping for Pre-Trained Model-Based Continual Learning
- Advancing Analytic Class-Incremental Learning through Vision-Language Calibration
- Continual Test-Time Domain Adaptation
- Disentangle Source and Target Knowledge for Continual Test-Time Adaptation
- Orthogonal Projection Subspace to Aggregate Online Prior-knowledge for Continual Test-time Adaptation
- Tent: Fully Test-Time Adaptation by Entropy Minimization
- Test-Time Classifier Adjustment Module for Model-Agnostic Domain Generalization
- Few-Shot Adaptation of Medical Vision-Language Models