Abstract
Self-adaptive training (SAT) is a unified algorithm that uses a model's own historical predictions to dynamically calibrate the training process. By replacing static, potentially noisy labels with an exponential moving average (EMA) of model outputs, SAT significantly improves robustness to label noise and enables high-performance self-supervised learning using only a single view of data—making it up to 2x faster than industry standards like BYOL or MoCo.
Introduction
Deep neural networks are notoriously "greedy" learners, prone to overfitting noise whether it manifests as incorrect labels in supervised settings or random artifacts in unsupervised data. Traditional fixes often involve complex multi-stage training or fragile early-stopping heuristics. Self-Adaptive Training (SAT) addresses this by asking: Can we design a single algorithm that handles both noise in labels and the lack of labels entirely?
Methodology: Algorithm 1
The core of SAT lies in maintaining a "running average" of model predictions for every image using an Exponential Moving Average (EMA).
- EMA Targets: Instead of fitting to a fixed target $y$, the model updates training targets by blending original labels with its own confident historical predictions.
- Supervised SAT: After an initial "warm-up" period, the model employs a sample re-weighting scheme, reducing the influence of samples where predictions are inconsistent with targets.
- Self-Supervised SAT: Targets are initialized as random noise. The model then predicts its own accumulated historical representations. Crucially, this requires only a single view of the image per training step, bypassing the need for computationally expensive multi-view strategies.
Robustness to Noise
Standard Empirical Risk Minimization (ERM) is highly sensitive to data corruption. As shown in Figure 2, ERM quickly overfits to noisy labels, Gaussian noise, and shuffled pixels, leading to a sharp divergence between training and validation accuracy. In contrast, SAT maintains stable validation performance, effectively filtering out the noise to focus on underlying patterns.
On CIFAR-10 with 40% label noise, SAT improved classification accuracy by up to 9% over state-of-the-art methods. On ImageNet with 40% noise, it outperformed standard training by 3%.
The "Blessing" of Predictions in SSL
A key insight of the paper is that even when training on 100% random noise, a model's early predictions capture useful structural information. SAT "magnifies" this signal. Figure 4 illustrates how the accumulated representation (blue line) consistently improves linear evaluation accuracy over hundreds of epochs, significantly outperforming fixed models or raw noise baselines.
This mechanism allows SAT to achieve 72.8% top-1 accuracy on ImageNet linear evaluation using only 200 epochs. Because it requires only a single view, it is 1.3x to 2x faster than MoCo and BYOL, which require multiple augmented views of the same image.
Key Results and Insights
- Bypassing Double Descent: SAT exhibits a single-descent curve, suggesting that the "second descent" in standard training (where test error drops again after an initial rise) may simply be an artifact of overfitting to noise.
- Selective Classification: In tasks where the model can choose to "abstain" from predicting when unsure, SAT achieved up to a 50% relative improvement in error rates.
Limitations and Future Work
While powerful, SAT introduces a storage overhead, requiring the maintenance of a target vector for every image (approx. 1.12GB to 4.47GB for ImageNet). It also introduces new hyperparameters, such as the momentum term ($\alpha$) and warm-up epoch ($E_s$). Future research may extend this self-adaptive principle to domains like Natural Language Processing (NLP) or Reinforcement Learning, where noisy signals are prevalent.
Summary updated