When a deployed vision model adapts to a stream of unlabeled data by training on its own predictions (pseudo-labels), every mistake it makes becomes a training target. That mistake makes the next round of predictions slightly worse, and the errors compound. This is the familiar collapse of continual test-time adaptation (adapting a pretrained model online, batch by batch, to a sequence of shifting domains). Current methods fight it by filtering which pseudo-labels to trust. They still let the model that produces the pseudo-labels change on every batch, so the feedback loop runs as many times as there are batches. A separate literature in reinforcement learning solves a structurally similar problem. In low-switching-cost learning, the policy that is acting is replaced only when the agent has collected substantially new information, measured by the growth of a feature covariance matrix. This cuts the number of policy changes from $T$ to $O(d\log T)$ without hurting regret.
We propose to treat the pseudo-label generator as that rarely switched policy. A frozen "teacher" labels incoming data, and a student trains continuously on those labels. The teacher is replaced by the student only when the stream has delivered a doubling of information. Crucially, that information is measured in the feature space of the frozen source model, not the adapting one. The bet is that confirmation bias compounds with the number of times the label source refreshes, not the number of gradient steps. If so, bounding refreshes by information content should lower long-horizon error, and should place refreshes where they matter, at domain shifts.
Core Claim
Two claims, one mechanistic and one algorithmic, can be tested separately.
The mechanistic claim is that, at a fixed number of gradient steps, late-stream error in continual test-time adaptation depends mainly on how many times, and when, the pseudo-label source is refreshed. Between refreshes the student is doing ordinary distillation from a fixed target, which cannot amplify its own errors.
The algorithmic claim is that a determinant-doubling trigger computed on frozen source features does better than four alternatives at the same refresh budget: fixed-interval refresh, slow moving-average teachers, and the shift heuristics in current use. It does so because it spends refreshes on genuinely new regions of input space.
The second claim depends on avoiding a documented failure. Beyond Information Gain benchmarks switching criteria in deep reinforcement learning. It finds that determinant triggers on neural features fire excessively: they score an average reward-threshold switching improvement of 2.33 on 56 Atari games, against 3.60 for a feature cosine-similarity trigger. Their diagnosis is that the features themselves move under gradient updates, so the determinant "grows" even when no new data arrives. Our design removes that cause directly. The source network never trains, so its features are a fixed map, and the linear-theory accounting that bounds doublings applies again. This is the specific difference from naively porting the rule from LSVI-UCB-RareSwitch, and the experiments test it head-on.
Research Program
Let $\phi(x)\in\mathbb{R}^d$ be the penultimate features of the frozen source model, passed through a fixed random projection to $d$ between 64 and 256. The projection keeps the bound on the number of doublings meaningful. Let $\Sigma_t=\lambda I+\sum_{\tau\le t}\phi(x_\tau)\phi(x_\tau)^\top$ accumulate over all test inputs seen so far, where $\lambda$ is a ridge regularizer. The deployed teacher $f_{\text{T}}$ makes predictions and produces pseudo-labels. A student $f_{\text{S}}$, initialized from the source model, takes a gradient step on every batch using the teacher's labels. The teacher is replaced by the student only when
$$\det(\Sigma_t)\ \ge\ (1+\eta)\,\det(\Sigma_{t_{\text{last}}}),$$
where $t_{\text{last}}$ is the time of the previous switch and $\eta$ sets how much new information is required. By the ellipsoid potential lemma used in LSVI-UCB-RareSwitch and LSVI-UCB++, the number of switches is at most $O(d\log(1+T/\lambda)/\log(1+\eta))$, regardless of how the student's weights move. Determinant ratios are tracked cheaply with rank-one updates of the inverse covariance.
The second ingredient borrows the double-weighting idea from DW-LSVI-UCB. That method proves that, under corrupted rewards, samples in poorly explored directions must be down-weighted so that corruption cannot dominate there. Our analogue is that an input's novelty $u(x)=\lVert\phi(x)\rVert_{\Sigma_t^{-1}}$ (how far it lies from directions already seen) predicts that the teacher's label is unreliable. The student's loss on $x$ is therefore weighted by $\min(1,\alpha/u(x))$, where $\alpha$ sets how quickly novel samples are discounted. This produces an intended asymmetry. Novel inputs advance the switch clock, but they pull only weakly on the gradient until the teacher has been refreshed on them. DW-LSVI-UCB deals with external, bounded corruption under linear dynamics. Our corruption is self-generated, and the theory there does not transfer. We use only the weighting rule, and we ablate it.
On the theory side, the target is modest. Take a linear classifier head on frozen features, pseudo-labels with error rate $\epsilon_k$ in teacher epoch $k$, and a student that converges within an epoch. We would like to show that $\epsilon_{k+1}$ depends on $\epsilon_k$ only through a contraction plus a term controlled by the new mass. Excess error would then grow with the number of epochs, which the trigger bounds, rather than with $T$. We state this as a goal; we do not promise it.
Baselines
The naive references are the unadapted source model and continual TENT and batch-norm adaptation. Continual Test-Time Domain Adaptation reports CIFAR-10-C average error of 20.7% for continual TENT and 20.4% for batch-norm adaptation. The source-only error on continual ImageNet-C is 82.0%, as reported by Less is More.
The strongest existing classification method is PLF from Less is More. It combines self-adaptive per-class confidence thresholds with class-prior alignment and updates on every batch. It reports mean error of 14.8% on CIFAR-10-C, 29.9% on CIFAR-100-C and 59.7% on ImageNet-C, against CoTTA's 16.2%, 32.5% and 62.7%. PLF is the strongest competitor because it attacks the same pseudo-label noise with a better filter rather than a better schedule. Our method is orthogonal to it, so the fair comparison is PLF against PLF with rare teacher switching. Both use the same backbones, the same 15-corruption severity-5 sequence and the same per-method tuning budget on a held-out corruption ordering. Beating it means a lower mean error than PLF's reported numbers in our reproduction, by more than seed variance, on at least CIFAR-100-C and ImageNet-C.
For segmentation, the strongest method is Hybrid-TTA. Its dynamic domain-shift detector chooses between adapter tuning and full tuning using prediction-correlation drops, and it reports 62.2 mIoU on Cityscapes-to-ACDC at 2.004 frames per second, against CoTTA's 58.6. It is the closest prior work on when to update. We replace its detector with our trigger, keep the rest of its pipeline, and compare at equal frames per second. Beating it means matching or exceeding 62.2 mIoU with fewer full-tuning events.
Three ablations isolate the key ingredient. The first is fixed-interval teacher refresh, tuned to produce the same number of switches as our trigger. The second is the same determinant trigger computed on the adapting student's features, which is the setting where Beyond Information Gain saw failure. The third is the cosine-similarity feature trigger from that paper. All three are matched in switch count and use identical student training. A fourth ablation removes the novelty weighting.
Why not just…?
Why not just refresh the teacher every $N$ batches? This is the main threat. If the mechanistic claim is true but domain shifts are evenly spaced, fixed-interval refresh at a matched count could match us. The theory in Provably Efficient RL under Adaptivity Constraints shows that fixed batching pays an additive penalty when information arrives unevenly, but that is a linear-MDP result. We settle this empirically by randomizing domain durations within the corruption sequence. If fixed-interval refresh ties us there too, the honest contribution narrows to the refresh-count finding, and we would report it that way.
Why not just use a slower moving-average teacher, as in Mean Teacher or CoTTA? A small momentum also slows how quickly the label source changes. But it blends the student's latest errors into the teacher on every step, and it cannot speed up at a domain shift. Errors therefore still enter the teacher continuously, which is the tight coupling Dual Student authors blame for teachers reinforcing their own past mistakes. We sweep the momentum and compare at equal "effective refresh rate", measured as the cumulative teacher parameter change. We also track pseudo-label error over time against ground truth to see whether errors creep in between shifts.
Why not just use the cosine-similarity trigger, which already beat determinants in deep RL? It works on representations that drift under training, it has no bound on the number of switches, and its advantage was shown with true rewards, not self-generated labels. It is included as a matched-count ablation. If it equals the frozen-feature determinant trigger, then what matters is computing the trigger on frozen features, not the choice of statistic. We would test that by also running cosine similarity on frozen features.
Why not just filter harder, as PLF does? Filtering decides which labels to trust but still refreshes the labeler on every batch. Its threshold benefits reportedly fade over long sequences because initial calibration bias persists (Less is More). We stack the two rather than compete, so a gain from adding our schedule to PLF is direct evidence that the schedule addresses something filtering does not.
Proposed Experiments
The main suite runs continual CIFAR-10-C, CIFAR-100-C and ImageNet-C, built from the corruptions in Hendrycks and Dietterich, with severity-5 sequences, plus Cityscapes-to-ACDC repeated over multiple cycles. We report error or mIoU, the number of switches, gradient steps, frames per second, and per-domain pseudo-label error measured against held-back ground truth. The ten-sequence diverse-shift protocol from PLF tests robustness to ordering.
The decisive check for the mechanism holds gradient steps fixed and varies only the number and placement of teacher refreshes (every batch, fixed interval, triggered), then plots late-stream error against refresh count. If the curves collapse onto refresh count, the claim holds. If they track gradient steps instead, it fails.
The decisive check for the trigger examines where switches land. On sequences with known shift times, we measure the fraction of switches falling within a short window after each shift. We expect it to be high for the frozen-feature trigger and near chance for the student-feature trigger. We also expect the student-feature trigger to fire far more often, reproducing the failure in Beyond Information Gain in this setting.
Repeated ACDC cycles test the question of recurring domains. Because $\Sigma_t$ never discards old data, a returning domain triggers nothing. That outcome is desirable if the student remembers the domain and harmful if the student has forgotten it. To find out which, we compare against a variant whose covariance gradually forgets old data, which gives up the switch bound.
References
- Beyond Information Gain: An Empirical Benchmark for Low-Switching-Cost Reinforcement Learning
- Provably Efficient Reinforcement Learning with Linear Function Approximation under Adaptivity Constraints
- Nearly Minimax Optimal Reinforcement Learning for Linear Markov Decision Processes
- Logarithmic Regret for Linear Markov Decision Processes with Adversarial Corruptions
- Continual Test-Time Domain Adaptation
- Less is More: Pseudo-Label Filtering for Continual Test-Time Adaptation
- Hybrid-TTA: Continual Test-time Adaptation via Dynamic Domain Shift Detection
- Benchmarking Neural Network Robustness to Common Corruptions and Perturbations