A model deployed on a video feed, a car, or a wearable sensor has to keep adapting to new conditions without labels. The input also arrives in long runs of the same thing: hundreds of frames of one object, or hours of a person sitting still. Methods that adapt the model at test time cope with these runs by keeping a small memory of past inputs and rebalancing it so that each update sees a mix of classes. They decide what each stored input "is" from the model's current guess. That guess is the weak point. During a long run the model drifts toward the class it keeps seeing, so its confident guesses become systematically wrong, and the memory meant to protect against the run gets filled with mislabeled examples.
The training-dynamics literature has a known remedy for unreliable instantaneous confidence: watch how a sample's prediction behaves over many model updates, and trust the samples whose predictions stay put. Transplanting that remedy naively fails here, for a reason specific to correlated streams. A prediction can be "stable" only because the model has been pushed toward the same class for the last thousand steps. This note proposes a stability score for buffered test samples that counts a prediction as evidence only to the extent that it held while the model was being pulled toward other classes. That score then governs which stored samples receive pseudo-labels, how the memory is class-balanced, and which samples are evicted.
Core Claim
In one-pass test-time adaptation, each stream sample is seen once, so the stability signals of semi-supervised learning have no trajectory to measure. The exception is samples held in a replay buffer, and buffers are standard under temporal correlation. Every buffered sample is re-predicted whenever the buffer is used for an update. This gives it a trajectory across model states, the same object that SELFIE, MarginMatch and TC-SSL exploit across training epochs.
Our claim is that, under temporal correlation, this trajectory must be reweighted by the model's recent class bias. A margin observed while the model was predicting class $c$ almost everywhere says little about whether a stored sample truly belongs to $c$. A margin that survives a burst of class $d$ says a lot. Concretely, for a buffered sample $x$ with running label $c$, let $m_t(x)$ be its pseudo-margin at update $t$: its logit for $c$ minus its largest other logit, as in MarginMatch. Let $q_t$ be the momentum-tracked frequency of predicted classes over the recent stream, the same vector DELTA uses for reweighting. We define the burst-discounted stability
$$S(x) = \frac{\sum_t (1 - q_t[c])\, m_t(x)}{\sum_t (1 - q_t[c])},$$
which down-weights observations made when the model was already biased toward the sample's own label. The prediction is falsifiable. As correlation strengthens, plain stability and frame-to-frame consistency should lose their ability to separate correct from incorrect pseudo-labels, while $S$ should keep it. If $S$ does not beat plain trajectory stability on that diagnostic, the idea fails.
Relation to Prior Work
Two kinds of prior work motivate the proposal. The first is the correlated-stream adaptation methods, which document the failure. NOTE normalizes each instance against source statistics and balances a reservoir buffer using instantaneous predictions. RoTTA admits buffer samples by category balance, recency and instantaneous entropy, with an averaged teacher model supplying pseudo-labels. TRIBE routes features to per-class normalization statistics by pseudo-label and anchors the adapting model to the frozen source model. DELTA reweights the loss by predicted class frequency. All four trust a per-step label to decide balance. None asks whether that label has survived the model being pushed elsewhere, and TRIBE explicitly notes the corruption that follows when a burst of class $A$ is read as $B$.
The second kind supplies the method. MarginMatch and TC-SSL provide the trajectory statistics: an averaged margin and a measure of how much the output distribution fluctuates. Their setting is a fixed, shuffled unlabeled pool trained over many epochs, so the model's class bias does not swing with the data order. That is exactly the confound we correct.
The closest work is TTAC. It filters samples by temporal consistency of predictions before updating anchored feature clusters, but it is evaluated on sequential corruption streams rather than bursty label correlation and does not condition consistency on the model's current bias. TempT also constrains our design. It median-filters predictions across neighboring frames to build targets, which works for smoothly varying facial expressions. The same smoothing would reward the model for agreeing with itself across near-duplicate frames, and that agreement is exactly what correlated streams make uninformative.
Research Program
The work has three parts. The first is a diagnostic study, independent of any new method. On CIFAR-10-C and CIFAR-100-C streams ordered by a Dirichlet class distribution with a sweep of concentration values (from near-i.i.d. to single-class runs), we log for every buffered sample several candidate reliability signals:
- instantaneous softmax confidence;
- averaged-teacher confidence;
- agreement with neighboring frames;
- plain averaged margin;
- fluctuation in the style of TC-SSL;
- $S$.
We report how well each signal ranks correct over incorrect pseudo-labels, measured by area under the ROC curve, as a function of correlation strength. This tells us whether the confound is real before anything is built on it.
The second part is the method. We start from NOTE's architecture (instance-aware normalization, adapting only normalization scale and shift) and change three things. Each buffer slot stores a per-class margin average updated at no extra forward cost, because the buffer is already forwarded at every update. A newly admitted sample has no trajectory, so it enters on probation: it counts toward time-uniform sampling but not toward class balance, and it carries no pseudo-label loss until its accumulated weight $\sum_t (1-q_t[c])$ passes a threshold. Class balancing and eviction then use the stability-confirmed label, and within an over-represented class the sample with the lowest $S$ is evicted first.
The third part is a portability test. We plug the same score into RoTTA's admission rule and into TRIBE's routing of per-class normalization statistics, replacing instantaneous labels. This asks whether the ingredient helps beyond one host method. The added memory is one vector of class scores per slot, and the added compute is negligible.
Baselines
The naive reference is the unadapted source model, alongside TENT-style entropy minimization. TENT is the documented collapse case: NOTE reports 76.4% error for TENT on non-i.i.d. CIFAR-10-C, against 21.1% for NOTE and 36.2% for LAME. Any method must sit far from TENT and must not fall below the source model at high correlation.
The direct host and main competitor is NOTE. It is the method whose instantaneous-label balancing we replace, and its 21.1% non-i.i.d. CIFAR-10-C error (NOTE) is the reference number on that protocol. We rerun it from scratch with identical streams, buffer size, adapted parameters, optimizer and per-method tuning budget. Beating it means lower error at matched buffer size, with the gap growing as correlation strengthens.
TRIBE is the strongest method in the file under severe imbalance. It reports 19.16% error on CIFAR-10-C at 200× imbalance, against 32.45% for RoTTA (TRIBE). It costs three networks, so we compare at reported settings and also report memory and latency. Matching TRIBE's error with one network would count as a win. Strictly beating it requires the TRIBE-plus-$S$ variant to improve on TRIBE itself.
DELTA is the reference on dependent streams. It reports 64.1% accuracy on CIFAR-100-C in the dependent, class-balanced setting and 36.7% on ImageNet-C in the dependent, class-imbalanced setting (DELTA). RoTTA and UnMix-TNS are run under their own protocols and ours. Because published protocols differ in Dirichlet parameters and stream order, all headline comparisons use our reruns, and published numbers are used only as a sanity check that the reruns are faithful.
The ablations isolate the key ingredient. First, plain averaged margin with no burst discount, so that every weight $(1-q_t[c])$ is 1. Second, re-predicting the buffer with the latest model and balancing on that label, with no history at all. Third, balancing on the averaged teacher's label. The proposal's specific contribution is the gap between $S$ and the first ablation.
Why not just…?
Why not just use the averaged teacher's prediction, as RoTTA does? The teacher averages model weights over time, which smooths noise. During a long burst, however, the student drifts toward the burst class and the teacher drifts with it, only more slowly. Its confidence carries the same bias with a lag. This is a real threat for short bursts, where the lag may be enough. The diagnostic sweep settles it: if the teacher's separation of correct from incorrect labels matches $S$ at strong correlation, the extra machinery is unjustified and we will say so.
Why not just apply MarginMatch-style averaged margins to the buffer as is? This is the most natural transplant and the main ablation. Our argument is that in a correlated stream, consecutive model states share a class bias. Plain averaging therefore rewards samples that agree with the current burst, whether or not they are correct. If plain averaging performs as well as $S$, the core claim is wrong. The misclassified-burst stress test below is designed to separate the two.
Why not just check agreement across neighboring frames, as TempT and TTAC do? Neighboring frames in a correlated stream are near duplicates, so they agree almost regardless of correctness. TempT's own setting assumes smooth state changes, and it never faces a confidently wrong run. We expect frame agreement to show the steepest loss of separating power as correlation grows. The diagnostic study measures this directly rather than assuming it.
Why not just re-label the buffer with the current model at every step? This is cheap and removes stale labels. It uses the most-biased model state available, though, so it should be worst during long bursts. It is included as an ablation.
Proposed Experiments
The decisive check is the diagnostic curve. It plots, against Dirichlet correlation strength on CIFAR-10-C and CIFAR-100-C at severity 5, the area under the ROC curve for each signal's ranking of correct over incorrect pseudo-labels in the buffer. The proposal survives only if $S$ stays flat while plain averaging and frame agreement decline.
A second check is a constructed misclassified-burst stream. A long run of one class appears under a corruption that the source model systematically maps to another class. We track how pure the buffer's labels stay over time, the recall of the affected class after the burst ends, and whether TRIBE-style per-class statistics stay clean when routing uses $S$.
End-to-end evaluation covers three settings. On CIFAR-10-C, CIFAR-100-C and ImageNet-C under correlated and imbalanced streams, including 200× imbalance, we report error. On natural video, we report accuracy on YTBB-sub, where DELTA reports 75.7%, and on ImageNet-VID-C, where UnMix-TNS reports a 23% gain over test-time normalization but the file gives no absolute accuracy. On HARTH sensor streams we report accuracy, but no reported result was found on Lacuna for HARTH, so only our reruns are meaningful there. Throughout we also report per-class recall for classes absent from the current burst, buffer label purity, and latency per sample relative to a plain forward pass. The method is meant to add essentially nothing over NOTE, and a measurable slowdown would count against it.
References
- SELFIE: Refurbishing Unclean Samples for Robust Deep Learning
- MarginMatch: Improving Semi-Supervised Learning with Pseudo-Margins
- Time-Consistent Self-Supervision for Semi-Supervised Learning
- DELTA: DEGRADATION-FREE FULLY TEST-TIME ADAPTATION
- NOTE: Robust Continual Test-time Adaptation Against Temporal Correlation
- Robust Test-Time Adaptation in Dynamic Scenarios
- Towards Real-World Test-Time Adaptation: Tri-net Self-Training with Balanced Normalization
- Revisiting Realistic Test-Time Training: Sequential Inference and Adaptation by Anchored Clustering
- TempT: Temporal consistency for Test-time adaptation