When training models with limited annotations, we often turn to self-training: allowing a model to generate its own "pseudo-labels" on unlabeled data and then treating those predictions as ground truth.
However, standard self-training suffers from a fundamental flaw. If a model makes an incorrect prediction with high confidence, it updates its weights to reinforce that exact error. This creates a destructive positive feedback loop—known as confirmation bias—where the model rapidly diverges or collapses into trivial solutions.
To break this loop, a powerful mechanism has emerged across semi-supervised learning, test-time adaptation, and continual learning: Temporal Weight Averaging. Instead of having a model learn directly from its own immediate predictions, we maintain two models—an active student and a slowly evolving teacher—where the teacher's weights are updated as an Exponential Moving Average (EMA) of the student's weights.
The Geometry of Weight Averaging
To understand why temporal weight averaging works, we must look at the geometry of the loss landscape during optimization.
During training, the student model updates its parameters $\theta_{\text{student}}$ via backpropagation on a sequence of mini-batches. Because each mini-batch contains different samples, the optimization trajectory is noisy and prone to high-frequency oscillations. If we use the student model at any single instant to generate pseudo-labels, those labels inherit this high variance, leading to unstable targets.
Student Trajectory: θ_t-2 ---> θ_t-1 ---> θ_t (Oscillating, High Variance)
\ /
EMA Teacher: ▼ ▼
θ_EMA (Centered, Low Variance)
By maintaining a teacher model whose weights $\theta_{\text{teacher}}$ are updated via:
$$\theta_{\text{teacher}} \leftarrow \alpha \theta_{\text{teacher}} + (1 - \alpha) \theta_{\text{student}}$$
where $\alpha$ is a momentum hyperparameter close to $1$, we effectively apply a low-pass filter to the student's optimization trajectory. The teacher network acts as a temporal ensemble of the student's historical states. Geometrically, averaging weights across an optimization path tends to pull the parameters toward the center of a flatter, more robust local minimum. This flatter region yields predictions that are significantly more stable and less sensitive to small perturbations in the input data.
Managing the Temporal Distance
The core tuning knob of this architecture is the momentum coefficient $\alpha$, which dictates the "temporal distance" or memory half-life of the teacher.
As explored by Likhomanenko et al. in Kaizen: Continuously improving teacher using Exponential Moving Average for semi-supervised speech recognition, this temporal distance is critical to preventing model collapse. If the teacher updates too rapidly ($\alpha$ is too small), it quickly absorbs the student's immediate errors, collapsing back into standard self-training. Conversely, if it updates too slowly, the teacher lags behind, providing outdated and overly conservative guidance that stalls learning.
However, a static update rate can be problematic when a model encounters highly imbalanced data. To address this, Kar demonstrates in Revisiting Class Imbalance for End-to-end Semi-Supervised Object Detection that standard EMA updates suffer from a severe temporal lag during periods of rapid weight fluctuation. By modifying the update mechanism to place higher weight on recent student improvements, the teacher can adapt more responsively to rare classes without sacrificing overall stability.
Preventing Collapse in Non-Stationary Environments
The stabilizing properties of temporal weight averaging are especially vital when models must adapt online to changing environments without any human supervision.
In continual test-time adaptation, a model encounters a constantly shifting stream of data distributions (such as a vision model navigating changing weather conditions). The authors of Continual Test-Time Domain Adaptation (CoTTA) leverage an EMA teacher to provide stable pseudo-labels across these non-stationary streams, showing that the temporal ensemble prevents the catastrophic error accumulation that typically causes online models to forget their base knowledge.
This principle applies equally well to parameter-efficient fine-tuning. For instance, Ma et al. demonstrate in SwapPrompt: Test-Time Prompt Adaptation for Vision-Language Models that maintaining an EMA of prompt parameters provides a reliable reference point during inference-time updates. This dual-prompt system allows the model to absorb new domain knowledge via the student prompt while preserving historical context via the target prompt.
Gated and Adaptive Updates
The latest evolution in temporal weight averaging moves away from blind, step-wise updates toward conditional or gated averaging.
When a student model encounters an exceptionally noisy or ambiguous batch of data, its gradient updates can become erratic, threatening to pollute the teacher's parameters. To insulate the teacher from these shocks, Ran et al. propose an adaptive update framework in AdaSemiCD: An Adaptive Semi-Supervised Change Detection Method Based on Pseudo-Label Evaluation. By measuring the student's uncertainty and performance stability on the fly, the system dynamically limits or freezes the teacher's updates during periods of high student instability.
By treating the teacher as a protected repository of historical knowledge rather than a passive follower, gated temporal averaging ensures that the pseudo-labels guiding the system remain clean, reliable, and robust against the noise of real-world data streams.