Test-time adaptation lets a deployed classifier keep learning from the unlabeled inputs it sees, for example as a camera drifts into fog or night. The usual recipe, introduced by Tent, takes gradient steps that make the model's own predictions more confident. It updates only a small set of per-channel scale and shift parameters inside the normalization layers. This works until it fails. With a learning rate that is slightly too large, a batch of one, or a stream that keeps changing, the model can drift into predicting one class for everything. Current fixes add hand-set thresholds: entropy cutoffs, sharpness radii, and reset rules. Each must be tuned with labels the deployment never has.
We propose borrowing one piece of machinery from second-order optimization, the trust region, and making it work without labels. A trust region is a rule that fits a local quadratic model of the loss, takes the step that model recommends, and then checks whether the loss actually fell by the predicted amount. Each step's size comes from a curvature estimate computed on the incoming target data. After the step, we check the model's predicted drop in entropy against the drop actually realized on inputs the step never saw. When the two agree, the step is trusted. When they disagree, which is what happens when the update fits noise or pushes toward collapse, the method shrinks its steps automatically. The claim is that this single agreement ratio can replace the tuned learning rate, and can warn of collapse earlier than the signals current methods rely on.
Core Claim
The curvature we use is the Fisher information of the model's own predictive distribution. It is a positive semi-definite matrix $F = J^\top(\mathrm{diag}(\hat p) - \hat p\hat p^\top)J$, where $J$ is the Jacobian of the logits with respect to the adapted normalization parameters and $\hat p$ is the softmax output on the current test batch. It needs no labels. Along the proposed first-order direction $d_t$ (for example the Adam direction on the entropy loss), we set the step size and the predicted decrease as
$$\alpha_t = \frac{-g_t^\top d_t}{d_t^\top (F_t + \lambda_t I)\, d_t}, \qquad \rho_t = \frac{\mathcal{L}_{\text{held}}(\theta_t) - \mathcal{L}_{\text{held}}(\theta_t + \alpha_t d_t)}{\tfrac12 \alpha_t\,(-g_t^\top d_t)}.$$
Here $g_t$ is the entropy gradient on the batch used for the update, $\lambda_t$ is a damping term, and $\mathcal{L}_{\text{held}}$ is the entropy on inputs excluded from the update. Those inputs are either the other half of the batch or, when the batch size is one, the next arriving sample, whose forward pass at the new parameters is the inference we must run anyway. The denominator of $\rho_t$ is the decrease predicted by the quadratic model at the optimal step, which is where the factor of one half comes from. $\lambda_t$ grows when $\rho_t$ is small and shrinks when $\rho_t$ is near one, following the standard Levenberg–Marquardt rule.
The method comes from Adam through a Second-Order Lens (AdamQLR). AdamQLR grafts exactly this damping and quadratic step sizing onto Adam and reports that the untuned version matches tuned Adam. It was only tested on offline, supervised, stationary training, and it measures agreement on the same batch it steps on. In test-time adaptation that same-batch check is uninformative, because entropy on the batch you just fit will always drop. Moving the check to held-out stream inputs turns $\rho_t$ from a step-size heuristic into a test of whether the step generalizes. That is the new ingredient.
Research Program
The motivating evidence comes from three papers. SAR observed that gradient norms spike just before adaptation collapses. LCoTTA found that gradients from confidently wrong samples scatter in directions orthogonal to the low-dimensional subspace where useful gradients live. CAZO found that the Hessian of the adaptation loss is low-rank and slowly varying, with the top 20 eigenvalues carrying over 96% of the curvature. Together these suggest that harmful updates are ones the local geometry does not support, and that the gap between predicted and realized progress should reveal them.
The constraint comes from the hybrid-optimization literature. FUSE and related work report that quasi-Newton curvature breaks under mini-batch noise, which rules out an L-BFGS-style design on tiny test batches. Preconditioned gradient descent can lock a model into a lazy regime, which cautions against full Newton steps. We therefore keep the first-order direction and use curvature only to size the step along it. The cost is one forward-mode Jacobian-vector product to get $Jd_t$, a small quadratic form, and one extra forward pass for $\mathcal{L}_{\text{held}}(\theta_t)$. No extra backward pass is needed. This differs from EATA, whose Fisher is computed once on clean source data and used only as a penalty against drift. Ours is recomputed on target data at every step and sets step sizes.
The work has three parts. The first is the controller, including the rule for the batch-size-one case and a reset triggered when $\rho_t$ stays below a fixed fraction for several steps. The second is a diagnostic study of whether $\rho_t$ predicts collapse. The third is a wall-clock accounting. The file warns that test-time optimization is latency-bound (The Latency Bottleneck of Test Time Optimization), so any accuracy gain must be reported alongside measured per-image latency.
Baselines
The naive reference points are the unadapted source model and test-time normalization, which only re-estimates normalization statistics. Tent reports 49.9% ImageNet-C error for test-time normalization versus 44.0% for Tent. Tent itself is the first-order reference. It will be run from the official hyperparameters and also from a learning-rate grid spanning two orders of magnitude, so its sensitivity can be shown.
The strongest competitors differ by regime. SAR is the strongest for batch size one and label imbalance. It reports 34.5% ImageNet-C accuracy at batch size one on ResNet50-GN, where Tent and EATA collapse, and 58.0% versus Tent's 47.3% on ViT-Base under online label imbalance. EATA is the strongest for standard batches and lifelong use. It reports 65.0% severity-5 ImageNet-C error versus Tent's 71.6%, and keeps about 74% clean accuracy where Tent falls to 0%. It is available as mr-eggplant/eata. CoTTA is the continual reference, reporting 16.2% CIFAR10-C error. LCoTTA is the closest subspace-based stabilizer. The file gives no single accuracy number for LCoTTA on Lacuna, only that it stays stable where baselines fall toward 0%.
Fairness rules apply to every method. All methods share the source checkpoint, the stream order, and the adapted parameter set. Every baseline gets the same number of labeled validation corruptions for tuning; we use a small held-out set of corruption types not included in the test set. Our method gets zero tuning of step size and only a fixed initial damping value.
Beating a baseline has a specific meaning in each regime. At batch size one, we must match or exceed SAR's 34.5% without SAR's tuned entropy threshold and radius. In continual runs, we must show no collapse where Tent collapses. Across the learning-rate grid, our worst case must be better than the tuned baselines' worst case.
Three ablations isolate the key ingredient:
- Same-batch check. Measure $\rho_t$ on the update batch instead of held-out inputs. We expect this to lose the collapse warning.
- Source Fisher. Replace the target Fisher with EATA's static source Fisher.
- First-order line search. Replace the quadratic model with a backtracking line search on the held-out entropy, using a matched number of forward passes.
Why not just…?
Why not just tune the learning rate? Tuning requires labels on the target shift. Tent's and SAR's failures at batch size one and in continual streams happen at learning rates that are fine elsewhere. We will report an oracle per-shift learning rate as an upper bound. If our method gets within a point of the oracle across all regimes, that is the result. If a single global rate matches us everywhere, the proposal's motivation is wrong, and the grid experiment will show it.
Why not just use SAR's sharpness minimization and gradient-norm reset? SAR is the real threat at batch size one. Its perturbation is isotropic, and its reset reacts to a gradient spike that, by SAR's own account, arrives just before collapse. We will measure lead time directly: how many steps before accuracy drops each signal fires, and its detection score across induced-collapse runs. If $\rho_t$ gives no earlier warning than the gradient norm, we will report that as a negative result. The controller can also be stacked on SAR's direction, so we test SAR alone and SAR plus trust region.
Why not just a first-order line search on held-out entropy? This is the most dangerous alternative, because it keeps the held-out check and drops the curvature. The matched-forward-pass ablation settles it. The Fisher model should buy a good step in a single evaluation, where backtracking needs several. If the line search matches our accuracy and latency, the honest conclusion is that the held-out check matters and the second-order part does not, and we will say so.
Why not a FOSI-style Newton step in the top curvature subspace? FOSI speeds up training by taking Newton steps along the extreme Hessian eigenvectors. However, the entropy Hessian is indefinite, and test-time adaptation takes only a handful of steps per domain. That combination makes aggressive curvature steps likely to speed up collapse rather than prevent it. We include a FOSI variant built on the entropy Hessian to test this directly.
Proposed Experiments
ImageNet-C with ResNet-50 (batch normalization). Run at batch size 64, following the setup of Tent and EATA.
ImageNet-C with ResNet50-GN and ViT-Base. Run at batch size one, under online label imbalance, and on mixed-corruption streams, following SAR.
Lifelong runs. Use ImageNet-C and CIFAR-10-C/100-C with repeated corruption cycles, compared against CoTTA and EATA.
Beyond corruptions. Add ImageNet-R and Cityscapes-to-ACDC segmentation to check that the controller transfers beyond synthetic corruptions.
Metrics. We report accuracy or error and mean corruption error. We also report worst-case accuracy over the learning-rate or damping grid, and clean-set accuracy after the stream as a forgetting measure. For each image we report latency and peak memory, using the same GPU as SAR's 50,000-image timing (115 s for SAR versus 110 s for Tent on a V100).
Decisive checks. We will induce collapse on purpose, using large learning rates and batch size one, then compare $\rho_t$, gradient norm, and entropy as early-warning signals. We will run the three ablations above. We will show that the step sizes the controller picks track the oracle learning rate across corruption types.
The proposal succeeds if three things hold together: the held-out $\rho_t$ gives earlier collapse warning than the gradient norm, untuned accuracy matches or beats SAR and EATA in their own regimes, and the Fisher model beats a matched line search. Anything less is reported as it falls.
References
- Tent: Fully Test-Time Adaptation by Entropy Minimization
- Adam through a Second-Order Lens
- Towards Stable Test-time Adaptation in Dynamic Wild World
- Lifelong Test-Time Adaptation via Online Learning in Tracked Low-Dimensional Subspace
- Curvature-Aware Zeroth-Order Optimization for Memory-Efficient Test-Time Adaptation
- FUSE: First-Order and Second-Order Unified Synthesis in Stochastic Optimization
- On the Convergence Behavior of Preconditioned Gradient Descent Toward the Rich Learning Regime
- Efficient Test-Time Model Adaptation without Forgetting
- Continual Test-Time Domain Adaptation
- FOSI: Hybrid First and Second Order Optimization