Controlling Temporal Pseudo-Label Marginals for Stable Online Test-Time Adaptation

Rani Amar, Ofir LindenbaumarXiv 2026 · arXiv

TL;DR

SCALAR (Sinkhorn Class-Marginal Regularization) is a lightweight, plug-and-play module that prevents model collapse during Continual Test-Time Adaptation (CTTA). By using the Sinkhorn algorithm to enforce a balanced class distribution over a moving window of recent predictions, it stabilizes the adaptation process even under severe, non-stationary distribution shifts with negligible computational overhead.

What Problem Does This Paper Address?

In the real world, models often encounter data at test time that differs from their training data (distribution shift). Continual Test-Time Adaptation (CTTA) allows models to update themselves online using this unlabeled stream.

However, CTTA is notoriously unstable. Because models use their own predictions as "pseudo-labels" for supervision (self-training), a small initial bias can create a feedback loop. The model starts predicting fewer and fewer classes, eventually leading to drift (gradual error accumulation) or total collapse (where the model predicts only one or two classes for everything). While existing methods try to filter "reliable" samples, they don't explicitly control the overall distribution of classes the model is learning from over time.

Methods

The authors propose SCALAR, which introduces a temporal constraint on pseudo-labels. Its architecture is shown in Figure 1:

Figure 1: Overview of the SCALAR framework. White blocks denote the baseline teacher–student CTTA pipeline; green blocks denote the added SCALAR module. Given an input view xu for the teacher branch and an augmented view xv for the student branch, the teacher produces logits zu and probabilities qu = softmax(zu). Recent teacher logits are stored in a FIFO window and converted to probabilities P, which are balanced toward a target prior π via Sinkhorn to produce balanced targets ˜qu. The student produces logits sv and probabilities pv, and is optimized using a mixture of the baseline consistency loss and the SCALAR loss based on ˜qu. Gradients are stopped through teacher-derived targets, marked by sg, and teacher parameters are updated via EMA. Here Lcons ≡Lconsistency and Lbal = CE(sg[˜qu], pv).
  1. FIFO Logit Queue: The model maintains a small buffer (e.g., 1024 samples) of its most recent predictions (logits).
  2. Sinkhorn Balancing: It applies the Sinkhorn-Knopp algorithm to these logits. This mathematical "nudge" rescales the predictions so that, across the entire window, the class distribution matches a target prior (usually a uniform distribution).
  3. Balanced Targets: These adjusted, "balanced" soft targets are then used to calculate the loss for updating the model.
  4. Plug-and-Play: SCALAR doesn't change the model architecture. It can be added to existing CTTA frameworks like teacher-student pipelines or entropy-minimization methods (like TENT or EATA).

Datasets

The researchers evaluated SCALAR on standard corruption benchmarks and newly designed "skewed" streams to test robustness:

  • Standard Benchmarks: CIFAR-10-C, CIFAR-100-C, and ImageNet-C (all at severity 5).
  • Skewed Streams: They created custom versions of ImageNet-C where the class distribution is intentionally imbalanced and changes over time (Fixed-skew and Dynamic-skew).
  • Long-horizon Streams: Tests involving over 400,000 images to see if stability holds over extended periods.

Key Insights and Ideas

  • The "Effective Class" Diagnostic: The authors use $C_{eff}$ (the effective number of classes) as a "smoke detector" for model health. As seen in Figure 2, a sharp drop in $C_{eff}$ almost always precedes a spike in error and eventual collapse.
  • Temporal vs. Batch Balancing: Balancing classes within a single small batch is often impossible (the batch might not even contain all classes). By balancing over a temporal window, SCALAR provides a stable signal even when individual batches are highly skewed.
  • Stability over Specialization: Even if the true test data is skewed, forcing the model toward a uniform prior acts as a powerful regularizer that prevents the self-training feedback loop from spiraling out of control.
  • State-Dependent Benefit: SCALAR is most effective when the model is in danger. When adaptation is already stable, SCALAR has a minimal effect, but it acts as a safety net during aggressive updates or severe shifts.
Figure 2: Temporal contraction across adaptation regimes. Temporal evolution of error (top) and effective pseudo-label classes Ceff (bottom) in a teacher–student continual test-time adaptation setup with a ViT-B/16 backbone (ImageNet-C, severity 5, single-pass protocol). Here, Ceff denotes the effective number of classes in the teacher pseudo-label distribution. Left: Balanced stream (no skew) exhibits gradual contraction and late-stage drift. Middle: Fixed skew with rotating dominant classes (medium adaptation) results in stronger drift. Right: Dynamic skew with fast adaptation induces rapid contraction leading to collapse. Across regimes, SCALAR stabilizes class usage and mitigates the associated error growth.

Key Results

  • State-of-the-Art Performance: On ImageNet-C, SCALAR achieved a top-1 error of 36.1%, significantly outperforming the baseline (38.1%) and other established methods like CoTTA (54.8%) and TENT (51.0%).
  • Preventing Collapse: In "High Speed" adaptation scenarios (where the model learns quickly), the baseline collapsed in 2 out of 3 runs. SCALAR remained stable in every single run (see Figure 4).
  • Complementary to Existing Methods: When added to EATA, a method that already selects "reliable" samples, SCALAR reduced the collapse rate from 50% (4/8 runs) to 0% (0/8 runs) under aggressive settings (Figure 6).
  • Efficiency: The module is incredibly lightweight, adding only 10 MB of peak GPU memory and causing no measurable increase in per-step runtime.

Limitations and Future Work

  • Class Absence: If certain classes are truly absent from the test stream for a long time, a uniform prior might "over-regularize" and hinder the model's ability to specialize in the classes that are present.
  • Task Scope: The current study focuses on image classification. Future work is needed to see how SCALAR performs in other domains like object detection or semantic segmentation.
  • Adaptive Priors: The authors suggest that future versions could potentially "learn" or adapt the target prior ($\pi$) rather than relying on a fixed uniform one.

Summary updated

Abstract

Continual Test-Time Adaptation (CTTA) methods update models online using a stream of unlabeled data to overcome distribution shifts encountered at test time. However, CTTA is often unstable, as it can reinforce biases in early predictions and increasingly predict only a few classes, resulting in drift or even collapse. In this paper, we identify narrowing pseudo-label class usage as an indicator of CTTA instability and quantify this behavior. Most existing methods have attempted to address this issue by either selecting reliable adaptation samples or re-weighting the loss to encourage diverse pseudo-labels, but they do not explicitly enforce a class marginal over a temporal window. Instead, we propose Sinkhorn Class-Marginal Regularization (SCALAR), a simple plug-and-play module that controls the pseudo-label distribution over a short temporal window. SCALAR works by maintaining recent predictions in a buffer, balancing their class distribution via the Sinkhorn algorithm to match a specified target distribution, and using the balanced soft targets for adaptation. Without changing the model architecture, SCALAR maintains the per-step runtime and increases the peak allocated GPU memory by only 10 MB on ImageNet-C. Our method is particularly effective on skewed, non-stationary data with severe distribution shifts, where SCALAR significantly alleviates drift and collapse when combined with existing stabilization methods based on entropy or diversity.
BibTeX
@misc{amar2026controlling,
  author = {Rani Amar and Ofir Lindenbaum},
  title = {{Controlling Temporal Pseudo-Label Marginals for Stable Online Test-Time Adaptation}},
  year = {2026},
  eprint = {2611.12242},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.12242},
}
Figure