Frame Changes for Self-Training: Removing Runaway Directions in Online Adaptation

A model that adapts during deployment by training on its own predictions can slowly corrupt itself. A visual tracker that updates its picture of the target from its own boxes gradually absorbs the background. A classifier that minimizes its own uncertainty on unlabeled test images becomes confidently wrong. This is called model drift. Current defenses decide which samples to learn from (confidence gates, memory buffers) or how far to move overall (small learning rates, flat-minimum regularizers). We propose a third option. Find the few parameter directions in which the feedback loop between prediction and update is self-amplifying, and change only the coordinates in which the model generates its own labels so that those directions stop feeding back. Adaptation stays free everywhere else.

The idea comes from a well-understood problem in gravitational physics. A field coupled to spacetime curvature can pick up a negative effective mass-squared, called a tachyonic instability: small perturbations grow exponentially instead of oscillating. Hell showed that for a massive vector field coupled to the Ricci tensor, such runaway modes can be removed by a disformal metric redefinition. That is a change of geometry that adds a term built from the vector field itself, $\bar g_{\mu\nu} = g_{\mu\nu} - (\beta/M_{pl}^2)A_\mu A_\nu$. Unlike a uniform rescaling, it treats one direction differently from the rest. Our claim is that self-training drift is structurally the same kind of instability. It can be diagnosed as negative curvature of the closed-loop update and cured by the same kind of targeted, low-rank frame change applied to the pseudo-labeler. This does not give up the adaptation, which is the analogue of setting the coupling to zero.

Core Claim

Write the online update as $\theta_{t+1} = \theta_t - \eta \nabla_\theta \mathcal{L}(\theta_t; \hat y(\phi_t))$. Here $\theta$ are the adapted parameters, $\eta$ is the step size, and $\hat y(\phi)$ are pseudo-labels (or template/box targets) produced by a labeler with parameters $\phi$. In standard self-training $\phi_t=\theta_t$; in teacher-based methods $\phi_t$ is a lagged copy. Linearizing around the current state, the effective curvature seen by the update splits into two parts:

$$H_{\text{eff}} = H_{\text{fit}} + H_{\text{fb}}, \qquad H_{\text{fb}} = \frac{\partial^2 \mathcal{L}}{\partial \theta\, \partial \hat y}\,\frac{\partial \hat y}{\partial \phi}\,\frac{\partial \phi}{\partial \theta}.$$

The first term, $H_{\text{fit}}$, is the ordinary curvature of fitting fixed targets. The second, $H_{\text{fb}}$, comes from the targets moving when the model moves. Drift is a direction $v$ with $v^\top H_{\text{eff}} v < 0$: the feedback term overwhelms the fitting curvature, just as $m_{\text{eff}}^2 = m_0^2 + \xi R$ turns negative when the curvature coupling is large.

A caveat shapes the design. Merely preconditioning the gradient (changing the metric of the update step) cannot fix this. A positive-definite preconditioner rescales but never flips negative curvature. The physics analogue is also not a reparametrization of the update. It is a change of which metric matter couples to. Accordingly, we redefine the labeler's parameters as

$$\bar\phi_t = \phi_t - B_t\, v_t v_t^\top (\phi_t - \phi_0),$$

where $\phi_0$ are the source (pre-deployment) weights, $v_t$ is the current leading runaway direction, and $B_t \in [0,1]$ is chosen so that $v_t^\top H_{\text{eff}} v_t$ becomes non-negative. The adapted model can still move along $v_t$ if the fitting term asks it to. Its own movement along $v_t$ simply no longer changes the labels it trains on.

Research Program

The first component is a diagnostic: an online estimate of the leading eigenpairs of $H_{\text{eff}}$. We compute these with a few power-iteration steps of Hessian-vector products on a small recent-sample buffer, refreshed every $k$ updates. We will test whether the most negative eigenvalue rises ahead of accuracy collapse. The collapse cases are the documented TENT collapse on noisy streams in SoTTA and the tracker "death spirals" discussed around KeepTrack and DLST. If the eigenvalue does not lead the collapse, the premise is false and the project stops there.

The second component is the correction itself: the disformal labeler map above, with $B_t$ set from the eigenvalue estimate. Three results from the physics papers act as design constraints, not decoration.

First, Domènech et al. showed that constant disformal coefficients tuned on a homogeneous background fail once local inhomogeneities appear. The stream analogue is bursts of outliers against an average stream, so $B_t$ and $v_t$ must be state-dependent rather than fixed after warm-up. We test the constant version as an ablation.

Second, Hell and Obata showed that local disformal maps cure Ricci-type couplings but not full Riemann-type couplings. This predicts a failure mode: when the negative subspace of $H_{\text{eff}}$ is high-rank, a low-rank correction will not suffice. We will measure the rank of the negative subspace on each benchmark and report where the method breaks.

Third, Goon, Hinterbichler and Trodden noted that tachyons whose growth time exceeds the age of the universe may be tolerable. This gives a principled threshold: correct only modes whose e-folding time, $1/(\eta|\lambda|)$ in update steps, is shorter than the stream length. Leaving the slower modes alone avoids needlessly suppressing adaptation.

The third component treats momentum as an Ostrogradsky-type risk. Ostrogradsky's theorem says that equations with higher time derivatives carry extra modes whose energy is unbounded below, unless the higher-order structure is degenerate. Momentum optimizers and multi-step template recurrences make the update second-order in time. We will check whether momentum reintroduces runaway modes that the first-order analysis misses, by comparing the eigenvalues of the augmented state (parameters plus momentum buffer). We will also test whether applying the projection to the momentum buffer is necessary. This is analogous to how Goon et al. need a constraint on the extra variable to remove the extra mode. This component is a secondary hypothesis; the core claim stands or falls without it.

Baselines

The naive reference points are the source model with no adaptation and unfiltered entropy minimization (TENT). On CIFAR10-C under the noise scenario, SoTTA reports TENT at 52.1% accuracy. We will rerun both on the identical streams.

The strongest existing method on the classification side is SoTTA, reported in SoTTA: Robust Test-Time Adaptation on Noisy Data Streams. It combines high-confidence, class-balanced sample gating with entropy-sharpness minimization. It is the strongest competitor because it targets exactly the noisy-stream drift we study, and it already uses curvature, though isotropically. Its reported numbers are 80.0% on CIFAR10-C noise, 84.5% under adversarial attack, and 82.2% on benign streams. We will rerun SoTTA from its released configuration, using the same backbone, stream order, batch size, and a hyperparameter search budget equal to ours. Beating it means exceeding its rerun noise-stream accuracy while not falling below its benign-stream accuracy. We will also report our correction added on top of SoTTA, since the mechanisms may be complementary. CoTTA, a lagged-teacher method, is included at its reported 54.8% noise-stream accuracy and rerun.

On tracking, we attach the correction to the online-optimized target appearance model of KeepTrack (Learning Target Candidate Association to Keep Track of What Not to Track), reported at 67.1% AUC on LaSOT. We choose KeepTrack because its drift defense (confidence-based memory replacement) is a gate we can switch off and replace, giving a clean comparison. MixFormer's reported 70.1% LaSOT AUC (MixFormer) is cited as the ceiling for context, not as a like-for-like competitor. On Anti-UAV410, DLST reports 68.81% state accuracy (DLST); we will use this as the reference for small-target thermal drift.

The key ablation replaces the directional correction with an isotropic one of matched strength. Concretely, the labeler is pulled uniformly toward $\phi_0$ by the same total amount $\|\bar\phi-\phi\|$; this is the "conformal" analogue. If the isotropic version matches the directional one, the disformal ingredient contributes nothing. Further ablations use a random direction instead of $v_t$, freeze $B_t$ and $v_t$ after warm-up, and apply the correction to the gradient (preconditioning) instead of the labeler. We expect the preconditioning variant to fail, per the argument above.

Why not just…?

Why not just use a smaller learning rate or a regularizer toward the source weights? This is the most serious threat. Uniform damping slows every mode, including the healthy ones that carry the useful adaptation. The instability therefore arrives later rather than never, at the cost of adapting less. We make this precise with the matched-strength isotropic ablation. We will also trace accuracy against total parameter displacement for both methods. If isotropic damping reaches the same accuracy at the same displacement, we will say so and recast the contribution as the diagnostic alone.

Why not just gate out low-confidence samples, as SoTTA and KeepTrack do? Gating removes samples, but a confident sample can still push along a runaway direction. Confident-but-wrong predictions are exactly what an established distractor produces, and the trajectory-crossing failures acknowledged by KeepTrack arise in that regime. Gating and our correction act on different objects, so the fair test is SoTTA with and without the correction. Gains that disappear once gating is on would mean the runaway directions are already sample-selectable.

Why not just use a slowly updated teacher (an exponential moving average of the weights) or a fixed first-frame anchor? A lagged teacher delays the feedback but does not remove it. CoTTA, which uses one, still falls to 54.8% on noisy CIFAR10-C. A permanently fixed anchor removes feedback in every direction and inherits the ceiling that STMTrack describes for static templates. Our labeler map sits between these, frozen only along $v_t$. The random-direction and isotropic ablations test whether that selectivity matters.

Why not just rely on sharpness-aware minimization? Sharpness minimization flattens the fitting landscape. The runaway comes from $H_{\text{fb}}$, the moving-target term, which sharpness-aware updates do not model. We will measure the most negative eigenvalue of $H_{\text{eff}}$ under entropy-sharpness minimization alone. If it stays non-negative, our correction is redundant on that benchmark.

Proposed Experiments

The classification study uses CIFAR10-C, CIFAR100-C and ImageNet-C streams under the benign, near-OOD, far-OOD, uniform-noise and adversarial scenarios defined in SoTTA. We report accuracy, the time course of the leading eigenvalue, and wall-clock overhead. For ImageNet-C, no reported result was found on Lacuna, so all ImageNet-C numbers will come from our reruns. The tracking study uses LaSOT (AUC, precision, normalized precision), OxUvA (MaxGM) for disappearance-heavy streams, and Anti-UAV410 (state accuracy), with frames-per-second measured on one GPU type for all methods. This addresses the overhead concern raised for memory- and sharpness-based defenses.

Four checks decide the project. First, the eigenvalue diagnostic must predict collapse before accuracy drops, measured as lead time and area under the curve for predicting a later collapse, on streams where TENT is known to fail. Second, the directional correction must beat the matched isotropic ablation on noisy streams while matching it on benign ones. Third, the method should fail precisely where the negative subspace is high-rank, which is the Riemann-coupling prediction; failure there is informative rather than embarrassing. Fourth, on KeepTrack with its confidence gate replaced by our correction, LaSOT AUC should stay at or above the reported 67.1%, with fewer identity switches on sequences containing similar distractors. If the diagnostic works but the correction does not beat isotropic damping, the contribution narrows to a drift early-warning signal, which is still useful and clearly scoped.