Self-Adaptive Training: Bridging the Supervised and Self-Supervised Learning

Lang Huang, Chao Zhang, Hongyang Zhang

Abstract

Self-adaptive training (SAT) is a unified algorithm that uses a model's own historical predictions to dynamically calibrate the training process. By replacing static, potentially noisy labels with an exponential moving average (EMA) of model outputs, SAT significantly improves robustness to label noise and enables high-performance self-supervised learning using only a single view of data—making it up to 2x faster than industry standards like BYOL or MoCo.

Introduction

Deep neural networks are notoriously "greedy" learners, prone to overfitting noise whether it manifests as incorrect labels in supervised settings or random artifacts in unsupervised data. Traditional fixes often involve complex multi-stage training or fragile early-stopping heuristics. Self-Adaptive Training (SAT) addresses this by asking: Can we design a single algorithm that handles both noise in labels and the lack of labels entirely?

Methodology: Algorithm 1

The core of SAT lies in maintaining a "running average" of model predictions for every image using an Exponential Moving Average (EMA).

  1. EMA Targets: Instead of fitting to a fixed target $y$, the model updates training targets by blending original labels with its own confident historical predictions.
  2. Supervised SAT: After an initial "warm-up" period, the model employs a sample re-weighting scheme, reducing the influence of samples where predictions are inconsistent with targets.
  3. Self-Supervised SAT: Targets are initialized as random noise. The model then predicts its own accumulated historical representations. Crucially, this requires only a single view of the image per training step, bypassing the need for computationally expensive multi-view strategies.

Robustness to Noise

Standard Empirical Risk Minimization (ERM) is highly sensitive to data corruption. As shown in Figure 2, ERM quickly overfits to noisy labels, Gaussian noise, and shuffled pixels, leading to a sharp divergence between training and validation accuracy. In contrast, SAT maintains stable validation performance, effectively filtering out the noise to focus on underlying patterns.

Comparison of accuracy curves between ERM (top) and SAT (bottom) under various noise conditions.

On CIFAR-10 with 40% label noise, SAT improved classification accuracy by up to 9% over state-of-the-art methods. On ImageNet with 40% noise, it outperformed standard training by 3%.

The "Blessing" of Predictions in SSL

A key insight of the paper is that even when training on 100% random noise, a model's early predictions capture useful structural information. SAT "magnifies" this signal. Figure 4 illustrates how the accumulated representation (blue line) consistently improves linear evaluation accuracy over hundreds of epochs, significantly outperforming fixed models or raw noise baselines.

Evolution of linear evaluation accuracy during training, highlighting the strength of accumulated representations.

This mechanism allows SAT to achieve 72.8% top-1 accuracy on ImageNet linear evaluation using only 200 epochs. Because it requires only a single view, it is 1.3x to 2x faster than MoCo and BYOL, which require multiple augmented views of the same image.

Key Results and Insights

  • Bypassing Double Descent: SAT exhibits a single-descent curve, suggesting that the "second descent" in standard training (where test error drops again after an initial rise) may simply be an artifact of overfitting to noise.
  • Selective Classification: In tasks where the model can choose to "abstain" from predicting when unsure, SAT achieved up to a 50% relative improvement in error rates.

Limitations and Future Work

While powerful, SAT introduces a storage overhead, requiring the maintenance of a target vector for every image (approx. 1.12GB to 4.47GB for ImageNet). It also introduces new hyperparameters, such as the momentum term ($\alpha$) and warm-up epoch ($E_s$). Future research may extend this self-adaptive principle to domains like Natural Language Processing (NLP) or Reinforcement Learning, where noisy signals are prevalent.

Summary updated

Abstract

We propose self-adaptive training-a unified training algorithm that dynamically calibrates and enhances training processes by model predictions without incurring an extra computational cost-to advance both supervised and self-supervised learning of deep neural networks. We analyze the training dynamics of deep networks on training data that are corrupted by, e.g., random noise and adversarial examples. Our analysis shows that model predictions are able to magnify useful underlying information in data and this phenomenon occurs broadly even in the absence of any label information, highlighting that model predictions could substantially benefit the training processes: self-adaptive training improves the generalization of deep networks under noise and enhances the self-supervised representation learning. The analysis also sheds light on understanding deep learning, e.g., a potential explanation of the recently-discovered double-descent phenomenon in empirical risk minimization and the collapsing issue of the state-of-the-art self-supervised learning algorithms. Experiments on the CIFAR, STL, and ImageNet datasets verify the effectiveness of our approach in three applications: classification with label noise, selective classification, and linear evaluation. To facilitate future research, the code has been made publicly available at https://github.com/LayneH/self-adaptive-training.
BibTeX
@article{huang2024self,
  author = {Lang Huang and Chao Zhang and Hongyang Zhang},
  title = {{Self-Adaptive Training: Bridging the Supervised and Self-Supervised Learning}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2024},
  doi = {10.1109/tpami.2022.3217792},
}
Figure