Temporal Smoothing of Pseudo Labels and Prototypes

107 concepts88 papers1 proposal

In weakly supervised, semi-supervised, and unsupervised learning, models face a fundamental chicken-and-egg dilemma. Without ground-truth labels, a model must generate its own targets—either by predicting soft "pseudo-labels" for unlabeled samples or by clustering features to discover class "prototypes." The model then updates its weights based on these self-generated targets.

This feedback loop is inherently unstable. If a training batch contains noisy samples, or if early-stage feature representations are poor, the model's instantaneous predictions will fluctuate wildly. Updating the model on these volatile targets causes rapid oscillations, label switching, and confirmation bias, ultimately leading to training collapse.

To break this cycle, a powerful and intuitive mechanism has emerged across diverse domains: temporal smoothing. Instead of updating targets instantaneously based on the current training step, models maintain a running, historical memory of labels and prototypes. By forcing these latent variables to evolve gradually, temporal smoothing acts as a stabilizing anchor for optimization.


The Core Mechanism: Low-Pass Filtering the Training Signal

To understand why temporal smoothing works, consider how a model updates a latent variable—such as a class prototype or a pseudo-label vector—at training step $t$. An instantaneous update sets the variable directly to the current batch's estimate $\hat{X}_t$. In contrast, a temporally smoothed update uses an Exponential Moving Average (EMA):

$$X_t = \alpha X_{t-1} + (1-\alpha) \hat{X}_t$$

Here, $\alpha \in [0, 1)$ is a momentum coefficient, typically set close to $0.9$ or $0.99$.

If we unroll this recurrence relation, the state at step $t$ becomes a weighted sum of all historical estimates, where the influence of past steps decays exponentially:

$$X_t = (1-\alpha) \sum_{i=0}^{t} \alpha^{t-i} \hat{X}_i$$

In the language of signal processing, this equation is a first-order low-pass filter applied over the time domain of training iterations. High-frequency noise—such as batch-to-batch variance, transient feature fluctuations, and incorrect early-stage predictions—is filtered out. Only the low-frequency trend, which represents the true underlying semantic structure of the data, is allowed to pass through.


Stabilizing the Label Space

When a model must disambiguate partial labels or assign pseudo-labels to unlabeled data, relying on the network's immediate output is dangerous. Early in training, the network lacks the discriminative capacity to make accurate predictions, meaning initial pseudo-labels are highly corrupted.

Without Smoothing:
[Batch t]    --->  Erratic Pseudo-Label  --->  Violent Weight Update  ---> Instability
[Batch t+1]  --->  Shifted Pseudo-Label  --->  Violent Weight Update  ---> Oscillation

With Temporal Smoothing:
[Historical Memory] ---\
                        +---> Smoothed Target ---> Stable Weight Update ---> Convergence
[Current Batch t]   ---/

To prevent the model from prematurely committing to incorrect classifications, researchers apply temporal smoothing directly to the label space. For instance, in partial label learning, Wang et al. demonstrate in their framework, PiCO, that updating pseudo-targets via a temporal moving average provides significant robustness against early-stage label noise. This gradual evolution prevents the model from overfitting to false positives within the candidate label sets.

This concept extends naturally to cross-modal and domain adaptation settings:

  • Semi-Supervised Learning: In audio-visual source localization, Ma et al. apply an EMA to pseudo-labels across training steps. This prevents the training signal from oscillating wildly when model confidence is low, ensuring steady convergence.
  • Source-Free Domain Adaptation: When adapting a model to a new domain without source data, Yan et al. maintain pseudo-labels as a moving average of predicted probability distributions over time. This temporal smoothing of soft labels reduces the impact of target-domain noise and stops the model from oscillating between discrete class assignments for ambiguous inputs.
  • Label Disambiguation: In positive-unlabeled learning, Zhao et al. constrain label refinement updates using an EMA, preventing rapid shifts in pseudo-label assignments and ensuring the model converges toward a consistent global label distribution.

Stabilizing the Feature Space: Prototypes and Centroids

While smoothing the label space protects the model from bad targets, smoothing the feature space ensures that the geometric references used to generate those targets remain stable.

Class prototypes (or cluster centroids) represent the average embedding of a category in the latent space. If prototypes are calculated purely from local batch statistics, they shift erratically from iteration to iteration due to data imbalance or outliers.

By accumulating feature representations across training iterations, models create a stable global reference. Cao et al. leverage this in video anomaly detection by preserving cluster centers from previous iterations as memory-based priors. This consistent guidance ensures that the representation of "normal" behavior remains stable even when individual training batches are highly imbalanced.

We see this exact geometric stabilization across a wide variety of tasks:

  • Zero-Shot Learning: In generative zero-shot learning, human-defined text descriptions often clash with visual realities. Chen et al. allow semantic prototypes to evolve during training, but they apply a smooth moving average to the updates. This temporal smoothing prevents the generative network from collapsing due to sudden shifts in the conditional input space.
  • Sign Language Recognition: For continuous sign language recognition, Rao et al. update global class representations using a momentum strategy. This smoothly incorporates new feature information, preventing drastic shifts in the reference dictionary and maintaining a reliable memory of class features.
  • Object Re-Identification: In unsupervised object re-identification, Zhang et al. refine noisy pseudo-labels by looking at clustering consensus across generations. By using a momentum averaging formula that incorporates propagated labels from previous epochs, the model avoids overfitting to transient errors in the clustering process.

Conclusion

Whether applied to soft labels, cluster centroids, or semantic prototypes, temporal smoothing solves a fundamental problem in self-supervised and weakly supervised learning: it decouples the immediate output of a changing model from the targets used to train it. By treating training history as a low-pass filter, temporal smoothing ensures that representations evolve along a smooth, continuous trajectory, turning a potentially chaotic feedback loop into a stable path toward convergence.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_temporal_label_smoothing_a7c72bc7,
  title = {temporal label smoothing},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/temporal-label-smoothing/rdr_a7c72bc70a8f4284a13a65a14a687ae3},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_temporal_label_smoothing_a7c72bc7_v1,
  title = {temporal label smoothing},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/temporal-label-smoothing/rdr_a7c72bc70a8f4284a13a65a14a687ae3/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake