Contraction-Certified Damping for Continual Test-Time Adaptation

When a deployed model adapts itself to unlabeled data as it arrives, it trains on its own predictions. That loop can feed on itself. A slightly wrong prediction makes the model more confident in the wrong direction, which makes the next prediction more wrong, until the model outputs one class for everything. Current fixes for this are heuristics: randomly resetting weights, averaging a teacher model, or ranking losses. None of them can say in advance, for a given batch, whether the next update will stabilize or destabilize the model. This proposal treats each adaptation step as one iteration of a discrete dynamical system. It then asks the question stability theory asks of such systems: does this update map pull nearby parameter settings together (contraction) or push them apart?

The core idea is to measure, on every incoming batch, the most negative curvature of the self-training loss in the tiny parameter subspace being adapted. It then adds exactly enough pull back toward the source weights to make that one step a contraction, with no more pull than needed. Because the adapted subspace is small (normalization or prompt parameters, tens of thousands of numbers), this curvature can be estimated with a handful of Hessian-vector products. No formal verifier is required. The payoff is a tracking guarantee. The adapted parameters stay within a distance of the current domain's "ideal" setting that is proportional to how fast the domain drifts. When a domain recurs, the model returns to the same place, which is precisely the unpredictable-recurrence regime where existing methods degrade.

Core Claim

Collapse in entropy-based test-time adaptation is a local loss of contraction. The negative curvature of the self-training objective in confident directions exceeds whatever regularization is present. A per-step damping coefficient set from a measured lower bound on that curvature therefore prevents collapse while giving up less plasticity than any fixed regularizer. To state this concretely, let $\theta_k$ be the adapted parameters after $k$ batches, $\theta_0$ the source parameters, $\ell_k$ the unsupervised loss on batch $k$, $\eta_k$ the step size and $\mu_k \ge 0$ the damping. The update is

$$\theta_{k+1} = \theta_k - \eta_k\big(\nabla \ell_k(\theta_k) + \mu_k(\theta_k - \theta_0)\big).$$

Its local Jacobian is $I - \eta_k(H_k + \mu_k I)$, where $H_k$ is the Hessian of $\ell_k$ in the adapted subspace. This map is a contraction with rate $\rho_k < 1$ whenever

$$\mu_k > -\lambda_{\min}(H_k) + \epsilon \quad\text{and}\quad \eta_k < 2/(\lambda_{\max}(H_k) + \mu_k).$$

Here $\epsilon>0$ is a safety margin and $\lambda_{\min}, \lambda_{\max}$ are the extreme eigenvalues. Take $V_k = \|\theta_k - \theta_k^\star\|^2$ as the Lyapunov function (an "energy" that must shrink), where $\theta_k^\star$ is the fixed point of the damped update for the current domain. The standard contraction-tracking argument then bounds the steady-state tracking error by (drift of $\theta_k^\star$ per step + gradient noise)$/(1-\rho)$. The form comes from summing a geometric series of per-step contractions. The method sets $\mu_k$ to the smallest value meeting the condition, estimated by a few Lanczos iterations. It also inflates $\epsilon$ with the batch-to-batch variance of the estimate so that small batches, where the estimate is noisy, receive more damping.

Relation to Prior Work

The optimization literature supplies the method. Lyapunov analyses of discrete iterative algorithms handle drift and discretization error explicitly. Examples include the unified treatment of stored-gradient methods in A Short and Unified Convergence Analysis of the SAG, SAGA, and IAG Algorithms, the error-accumulation bounds in A Tight Theory of Error Feedback Algorithms in Distributed Optimization, and the cross-term constructions for inertial methods in On damped second-order gradient systems. All of these analyze a fixed objective with known curvature constants. We instead estimate the constants online, on a non-stationary, self-referential objective, and use them to choose the damping rather than only to certify it afterward.

Four papers already sit at the intersection, and each constrains the design. Control the GNN learns a neural Lyapunov function over graph features and verifies it with an SMT solver (a formal constraint checker). Its reported bottleneck is scale: it needed PCA-downsampled features and a linear backbone. That is why we avoid learned, verified certificates and use an analytic one that is checked numerically. Lyapunov-Guided Self-Alignment filters imagined rollouts by a Lyapunov decrease condition but keeps weights frozen. Our object is the weight trajectory itself. Flow Matching for Few-Trial Neural Adaptation with Stable Latent Dynamics bounds latent activations using Lyapunov exponents. It explicitly leaves the parameter-update dynamics uncertified, and that gap is the one we target. TEA: Test-Time Energy Adaptation minimizes free energy, which resembles dissipation. It carries no stability argument and was not tested under recurring shifts.

The continual test-time adaptation literature motivates the problem. Continual Test-Time Domain Adaptation (CoTTA) counters error accumulation with a mean teacher, 1% random weight restoration and heavy augmentation. ViDA routes updates between low-rank and high-rank adapters by dropout uncertainty. Ranked Entropy Minimization (REM) prevents collapse with a ranking loss over masked patches, and its authors state that why this works remains unproven. DPCore shows that ViDA degrades when domains recur unpredictably. Our method is a wrapper around any of these losses, not a competitor loss. Its claim is that the damping schedule, not the loss, is the missing stability ingredient.

Research Program

The first stage is diagnostic and can kill the idea cheaply. We run TENT (plain entropy minimization) on streams known to collapse, logging $\lambda_{\min}(H_k)$ in the normalization-parameter subspace alongside accuracy. If collapse is not preceded by $\lambda_{\min}$ turning sufficiently negative, with a lead time of tens of batches, the central mechanism is wrong and we stop.

The second stage builds the damped update with a Lanczos estimator using 5 to 10 Hessian-vector products per batch. It applies the variance-inflated margin and a warm-started eigenvector to cut cost. We apply it to TENT, TEA-style energy and REM losses. The third stage proves the tracking bound under stated assumptions: local validity of the quadratic model within a trust region, bounded estimation error in $\lambda_{\min}$, and bounded drift. It then empirically checks each assumption, including whether the adapted parameters stay inside the trust region in which the curvature estimate was taken. The fourth stage extends the analysis to momentum updates. There we borrow the heavy-ball cross-term Lyapunov function from An accelerated lyapunov function for Polyak's Heavy-ball on convex quadratics to derive the corresponding damping condition.

Baselines

The do-nothing references are the frozen source model and test-time batch-normalization statistic updating. On CIFAR-10-C, CoTTA's paper reports 20.4% error for batch-norm adaptation. These show whether any gradient-based adaptation is worth its risk at small batch sizes.

Plain TENT is the naive gradient baseline. REM reports 51.0% mean error for Tent on ImageNet-C, against 55.8% for the source model. It is the loss our damping most directly repairs, so TENT plus damping versus TENT alone is the cleanest demonstration.

The strongest existing methods are REM and DPCore. REM reports 39.2% mean ImageNet-C error in the continual setting and 9.4% on CIFAR-10-C. DPCore reports 42.1% on ImageNet-C in the continual dynamic-change setting, against 52.1% for ViDA. We rerun both from official code, including yunbeizhang/DPCore and the LayerwiseTTA pipeline. Each gets the same backbone, stream orders and batch sizes, and the same hyperparameter budget on a held-out corruption sequence that is disjoint from test corruptions. Beating them means REM plus damping improves on reproduced REM in the dynamic-change setting and at batch sizes of 1 to 4. It should not lose more than about 0.5 points at batch size 64.

CoTTA is included as the heuristic-restoration reference. We compare it both on accuracy and on compute per sample, since its augmentations are expensive.

Three ablations isolate the key ingredient. The first is a fixed anchor, which uses the same update with constant $\mu$ grid-searched on the held-out sequence. The second is a shuffled-curvature control, which applies the same average damping but with $\mu_k$ drawn from another batch's estimate. The third is certificate-only: we compute $\rho_k$ but do not enforce it, which tests whether monitoring alone predicts failure.

Why not just…?

Why not just add a fixed L2 pull toward the source weights? This is the most serious threat, because it is our method with $\mu$ held constant. The argument against it is that the needed damping varies by orders of magnitude across corruption types and batch sizes. A constant tuned for the worst case suppresses plasticity everywhere else. The fixed-anchor ablation, given an equal tuning budget, settles this. If a tuned constant matches the adaptive rule across both the sequential and dynamic-change settings, the adaptive part is not justified. The contribution would then shrink to a diagnostic explaining why anchors work, and we would report it that way.

Why not just lower the learning rate? REM documents that TENT collapses at slightly higher learning rates. However, a small step only slows the drift toward the collapsed state. It does not remove the negative-curvature direction, and it hurts recovery after domain switches. Long-horizon runs over 10 recurrence cycles of Cityscapes-to-ACDC and ImageNet-C show the difference. A small-step TENT should degrade late, while a contracting update should stay flat. CoTTA's paper reports 58.6 mIoU on the repeated ACDC sequences, and ViDA's paper reports 61.1 to 62.3 across 10 rounds, as reference trajectories.

Why not just detect collapse and reset to source? Resetting is cheap and needs no Hessian products. But it discards useful adaptation and reacts only after damage has occurred. The certificate-only ablation doubles as a reset trigger: reset when $\rho_k$ exceeds 1. We compare it against a confidence-based reset trigger and against enforced damping. If reset-on-certificate performs as well as enforcement, the honest conclusion is that curvature is a good alarm but damping adds nothing.

Why not learn a Lyapunov function, as in Control the GNN? That paper's SMT verification needed PCA and a linear backbone, which rules it out for ViT-scale continual adaptation. Our analytic contraction condition trades a formal certificate for one checked numerically at each step. That trade is stated openly.

Proposed Experiments

The benchmarks are CIFAR-10-C, CIFAR-100-C and ImageNet-C. Each is run in both the fixed-order continual setting and DPCore's dynamic-change setting with unpredictable recurrences, at batch sizes 1, 4, 16 and 64. Cityscapes-to-ACDC over 10 cycles tests long-horizon stability. Metrics are mean error or mIoU, expected calibration error (following TEA's use of calibration under shift), the worst-window error over the stream, and wall-clock time per sample.

Several checks are decisive. Curvature must predict collapse ahead of time in the diagnostic stage. REM plus damping must improve on reproduced REM in the dynamic-change and small-batch regimes. Adaptive damping must beat the tuned fixed anchor, and the shuffled-curvature control must lose that advantage. Finally, measured parameter distance to the per-domain fixed point must scale with drift rate as the tracking bound predicts. We will test this by varying how many batches each domain lasts in the stream and plotting steady-state distance against $1/\text{duration}$.