Curriculum Learning by Optimizing Learning Dynamics

Tianyi Zhou, Shengjie Wang, Jeff A. BilmesAISTATS 2021 · OpenReview

Introduction

Standard machine learning typically relies on random mini-batch Stochastic Gradient Descent (SGD), where a model repeatedly processes a static dataset. Curriculum Learning (CL) seeks to improve this by dynamically selecting training data, mimicking human learning patterns. However, traditional CL methods often depend on human-designed heuristics—such as prioritizing "easy" or "hard" samples—which can be contradictory and are not directly derived from the optimization objective. This paper introduces Dynamics-optimized Curriculum Learning (DoCL), a mathematically principled approach that selects data based on its ability to accelerate learning dynamics toward the ground truth.

Methodology: Dynamics-optimized Curriculum Learning (DoCL)

The core of DoCL is a scoring function derived from the mathematics of optimization dynamics. Rather than relying on loss-based heuristics, the authors formulate data selection as an optimization problem: at each step, select the subset of data that maximizes the rate at which the model's predictions for the entire distribution move toward their true labels.

The Scoring Function

The derived per-sample score is the inner product of two critical components:

  1. The Residual: The distance between the current prediction and the ground truth.
  2. Linear Dynamics: The rate at which the prediction changes under gradient flow on the overall data distribution.

This score identifies samples that are both "hard" (high residual) and "representative" (gradients align with the broader dataset).

Implementation Details

To ensure practical efficiency, DoCL estimates these dynamics using byproducts of standard training, resulting in negligible computational overhead. The algorithm employs an Exponential Moving Average (EMA) for score stability and utilizes cyclical learning rates to improve score estimation. Unlike traditional CL, which starts with small subsets, DoCL begins with the full dataset and gradually reduces the subset size as the model converges.

Test accuracy comparison on CIFAR-10 when training on samples with different scores.

Theoretical Insights

By analyzing the scoring function through the lens of the Neural Tangent Kernel (NTK), the authors demonstrate that DoCL is naturally robust to outliers and noisy labels. While an outlier may have a large residual, its gradient typically lacks alignment with the rest of the data distribution, leading to a low DoCL score and subsequent exclusion from the curriculum.

Experimental Results

The authors evaluated DoCL across nine image classification datasets, including CIFAR-10/100, ImageNet, and fine-grained datasets like Food-101 and Birdsnap.

Test set accuracy vs. training batches across four major datasets.

DoCL achieved state-of-the-art accuracy across all benchmarks. Notable improvements include:

  • ImageNet: Top-1 accuracy increased from 75.04% (Random SGD) to 79.54%.
  • CIFAR-100: Accuracy improved from 79.64% to 83.23%.
  • Fine-Grained Classification: On datasets like Food-101 and Birdsnap, DoCL trained from scratch achieved results competitive with models typically requiring pre-training.

Conclusion

Dynamics-optimized Curriculum Learning (DoCL) provides a rigorous mathematical foundation for data selection in machine learning. By moving beyond heuristics and focusing on the acceleration of learning dynamics, DoCL significantly enhances training efficiency and final model performance across diverse tasks. Future work may explore optimizing dynamics over the entire training trajectory rather than just instantaneous steps.

Summary updated

BibTeX
@inproceedings{zhou2021curriculum,
  author = {Tianyi Zhou and Shengjie Wang and Jeff A. Bilmes},
  title = {{Curriculum Learning by Optimizing Learning Dynamics}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2021},
  url = {https://openreview.net/forum?id=mTeWdj8lVoc},
}
Figure