On The Power of Curriculum Learning in Training Deep Networks

Guy Hacohen, Daphna Weinshall

Abstract

Training neural networks is traditionally done by providing a sequence of random mini-batches sampled uniformly from the entire training data. In this work, we analyze the effect of curriculum learning, which involves the non-uniform sampling of mini-batches, on the training of deep networks, and specifically CNNs trained for image recognition. To employ curriculum learning, the training algorithm must resolve 2 problems: (i) sort the training examples by difficulty; (ii) compute a series of mini-batches that exhibit an increasing level of difficulty. We address challenge (i) using two methods: transfer learning from some competitive "teacher" network, and bootstrapping. In our empirical evaluation, both methods show similar benefits in terms of increased learning speed and improved final performance on test data. We address challenge (ii) by investigating different pacing functions to guide the sampling. The empirical investigation includes a variety of network architectures, using images from CIFAR-10, CIFAR-100 and subsets of ImageNet. We conclude with a novel theoretical analysis of curriculum learning, where we show how it effectively modifies the optimization landscape. We then define the concept of an ideal curriculum, and show that under mild conditions it does not change the corresponding global minimum of the optimization function.

Introduction

Standard deep learning training protocols typically rely on uniform random sampling of mini-batches. While effective, this approach ignores the potential benefits of structured data presentation. Inspired by human education, where students progress from simple arithmetic to complex calculus, curriculum learning (CL) proposes training models on "easy" examples before gradually introducing "harder" ones.

The implementation of CL in deep learning faces two primary hurdles: the Scoring Problem (defining and ranking image difficulty automatically) and the Pacing Problem (determining the optimal rate at which to introduce harder data). This paper investigates automated solutions to these challenges and provides a theoretical foundation for why CL enhances the training of Convolutional Neural Networks (CNNs).

Methodology

Scoring Functions

To rank images by difficulty without manual labeling, the researchers propose two automated methods:

  1. Transfer Scoring: A pre-trained "teacher" network (e.g., Inception or ResNet trained on ImageNet) extracts features from the target dataset. A simple classifier trained on these features provides confidence scores, which serve as a proxy for image "easiness."
  2. Self-Taught (Bootstrapping): The model undergoes a brief initial "vanilla" training phase. Its own confidence scores on the training set are then used to rank examples for a subsequent curriculum-based training run.

Pacing Functions

The pacing function determines how the fraction of available training data grows over time. The researchers explored several strategies to manage this growth, including fixed exponential growth, varied exponential growth, and single-step transitions.

Comparison of different pacing functions used to increase the fraction of training data over iterations.

Empirical Results

The effectiveness of CL was evaluated on CIFAR-10, CIFAR-100, and subsets of ImageNet. The results consistently demonstrated that CL is particularly beneficial for difficult tasks.

Accuracy and Convergence

Curriculum learning significantly accelerates convergence and often leads to higher final test accuracy. For instance, on CIFAR-100 tests, CL achieved approximately 40% accuracy compared to 38% for vanilla training. On CIFAR-10 with VGG architectures, accuracy improved from ~91% to ~92%.

Training progress comparing curriculum learning against vanilla, random, and anti-curriculum baselines.

The Importance of Order

A critical finding is the negative impact of an "anti-curriculum"—training on the hardest examples first. This approach significantly degrades performance, confirming that the specific sequence of data presentation is a vital hyper-parameter in deep learning. Random sampling, while better than anti-curriculum, still lags behind the structured curriculum approach in both speed and final performance.

Theoretical Analysis

The authors provide a novel proof characterizing CL as a Bayesian prior that modifies the optimization landscape. While CL does not change the location of the global minimum under mild conditions, it makes the landscape steeper and more direct. This "guiding" effect helps the model's weights converge more efficiently and often leads them toward more generalizable regions of the parameter space.

Conclusion

Curriculum learning offers a robust framework for improving both the speed and final performance of deep networks. By leveraging automated scoring via transfer learning or bootstrapping, researchers can implement effective curricula without human intervention. Future work remains in refining pacing functions to reduce hyper-parameter overhead and exploring the relationship between target-based ranking and self-paced learning.

Summary updated

Abstract

Training neural networks is traditionally done by providing a sequence of random mini-batches sampled uniformly from the entire training data. In this work, we analyze the effect of curriculum learning, which involves the non-uniform sampling of mini-batches, on the training of deep networks, and specifically CNNs trained for image recognition. To employ curriculum learning, the training algorithm must resolve 2 problems: (i) sort the training examples by difficulty; (ii) compute a series of mini-batches that exhibit an increasing level of difficulty. We address challenge (i) using two methods: transfer learning from some competitive ``teacher" network, and bootstrapping. In our empirical evaluation, both methods show similar benefits in terms of increased learning speed and improved final performance on test data. We address challenge (ii) by investigating different pacing functions to guide the sampling. The empirical investigation includes a variety of network architectures, using images from CIFAR-10, CIFAR-100 and subsets of ImageNet. We conclude with a novel theoretical analysis of curriculum learning, where we show how it effectively modifies the optimization landscape. We then define the concept of an ideal curriculum, and show that under mild conditions it does not change the corresponding global minimum of the optimization function.
BibTeX
@inproceedings{hacohen2019power,
  author = {Guy Hacohen and Daphna Weinshall},
  title = {{On The Power of Curriculum Learning in Training Deep Networks}},
  booktitle = {International Conference on Machine Learning},
  year = {2019},
  url = {https://openreview.net/forum?id=B1NFZ2ZubH},
}
Figure