Snapshot Ensembles: Train 1, Get M for Free

Gao Huang, Yixuan Li, Geoff Pleiss, Zhuang Liu, John E. Hopcroft, Kilian Q. Weinberger

TL;DR

Snapshot Ensembling is a clever training strategy that provides the performance boost of a model ensemble for the computational cost of training just a single network. By using a cyclic learning rate schedule, the method forces a single model to visit and "record" multiple local minima during one training run, which are then combined at test time to significantly reduce error rates.

What Problem Does This Paper Address?

In deep learning, "ensembling"—training multiple versions of a model and averaging their predictions—is a reliable way to boost accuracy and robustness. However, there is a catch: it is computationally expensive. If training one state-of-the-art model takes a week on a cluster of GPUs, training an ensemble of ten models takes ten weeks. This linear cost increase makes ensembling impractical for many researchers and production environments.

Standard vs. Cyclic LR Schedule

The authors address this "ensemble tax." They challenge the traditional view of optimization, which treats the path toward a single final minimum as the only goal. Instead, they argue that the various local minima a model passes through during training contain diverse, useful information that can be harvested without extra work.

Methods

The core innovation is a technique called Cyclic Cosine Annealing.

Standard training usually starts with a high learning rate that gradually decays to near zero. Once the learning rate is low, the model settles into a single local minimum, and training ends.

Snapshot Ensembling changes this:

  1. The Cycle: The learning rate follows a cosine wave. It starts high, drops to near zero to let the model converge to a local minimum, and then abruptly jumps back up to its initial high value.
  2. The Snapshot: Every time the learning rate hits its lowest point (the "trough"), the authors save a "snapshot" of the model's weights.
  3. The Escape: The sudden jump in the learning rate provides the model with enough "energy" to escape its current local minimum and search for a new, different one.
  4. The Ensemble: At test time, instead of using just the final model, the system averages the predictions (softmax outputs) of the $M$ snapshots collected during the run.
Training loss with Cosine Annealing

The total number of training epochs remains exactly the same as training a single standard model; the only difference is the schedule of the learning rate.

Datasets

The authors evaluated their method across several standard computer vision benchmarks:

  • CIFAR-10 & CIFAR-100: Small-scale natural image datasets.
  • SVHN (Street View House Numbers): A dataset of house numbers cropped from Google Street View images.
  • Tiny ImageNet: A subset of the massive ImageNet dataset, resized to 64x64.
  • ImageNet (ILSVRC 2012): The industry-standard large-scale classification benchmark.

They tested these datasets using various state-of-the-art architectures, including ResNet, Wide-ResNet, and DenseNet.

Key Insights and Ideas

  • Diversity via Disruption: The "aha moment" is that for an ensemble to work, the models must be diverse (i.e., they should make different mistakes). By abruptly raising the learning rate, the authors ensure the model doesn't just fall back into the same hole it just left, but finds a distinct region of the parameter space.
  • Local Minima are "Good Enough": The paper shows that while these intermediate snapshots might not be better than a fully converged single model individually, their collective wisdom is superior.
  • The Error Surface is Rugged: Through visualization (linear interpolation between snapshots), the authors demonstrate that their snapshots occupy truly different basins. If you try to move directly from one snapshot to another in a straight line, the error spikes, proving they are separated by high "ridges" in the loss landscape.

Key Results

Snapshot Ensembling consistently outperformed single-model baselines across all architectures and datasets at no additional training cost:

  • CIFAR-100: Using a DenseNet-100, the error rate dropped from 19.25% (single model) to 17.41% (Snapshot Ensemble).
  • CIFAR-10: The same architecture saw an error reduction from 3.74% to 3.44%.
  • ImageNet: On the much larger ResNet-50, a 2-cycle snapshot ensemble reduced the top-1 error from 24.01% to 23.33%.
  • Comparison to Baselines: The method outperformed other "cheap" ensembling tricks like Dropout or taking snapshots during a standard (non-cyclic) decay.

Limitations and Future Work

  • Test-Time Latency: While training is "free," you still have to run $M$ models during inference. This increases the computational cost at test time, which might be a bottleneck for real-time applications like self-driving cars.
  • Cycle Hyperparameters: The performance depends on choosing the right number of cycles ($M$). If cycles are too short, the model doesn't have enough time to converge to a "good" minimum before being kicked out again.
  • Future Directions: The authors suggest that Snapshot Ensembles could be combined with traditional ensembles (e.g., training 2 separate snapshot runs to get $2 \times M$ models) to push accuracy even further if more budget is available.

Summary updated

Abstract

Ensembles of neural networks are known to be much more robust and accurate than individual networks. However, training multiple deep networks for model averaging is computationally expensive. In this paper, we propose a method to obtain the seemingly contradictory goal of ensembling multiple neural networks at no additional training cost. We achieve this goal by training a single neural network, converging to several local minima along its optimization path and saving the model parameters. To obtain repeated rapid convergence, we leverage recent work on cyclic learning rate schedules. The resulting technique, which we refer to as Snapshot Ensembling, is simple, yet surprisingly effective. We show in a series of experiments that our approach is compatible with diverse network architectures and learning tasks. It consistently yields lower error rates than state-of-the-art single models at no additional training cost, and compares favorably with traditional network ensembles. On CIFAR-10 and CIFAR-100 our DenseNet Snapshot Ensembles obtain error rates of 3.4% and 17.4% respectively.
BibTeX
@inproceedings{huang2017snapshot,
  author = {Gao Huang and Yixuan Li and Geoff Pleiss and Zhuang Liu and John E. Hopcroft and Kilian Q. Weinberger},
  title = {{Snapshot Ensembles: Train 1, Get M for Free}},
  booktitle = {International Conference on Learning Representations},
  year = {2017},
}
Figure