A Simple Yet Powerful Deep Active Learning With Snapshots Ensembles

Seohyeon Jung, Sanghyun Kim, Juho LeeICLR 2023 · OpenReview

Overview

Active Learning (AL) helps machine learning models decide which data is most worth labeling, a crucial capability given that deep learning is notoriously data-hungry and expert labeling is expensive. To select the most "informative" samples from an unlabeled pool, models typically look for predictive uncertainty—essentially picking the samples they are most confused about.

However, deep neural networks are often poorly calibrated and struggle to know when they are wrong. To accurately estimate uncertainty, researchers traditionally rely on "Deep Ensembles" (training multiple models from scratch) or "Monte-Carlo (MC) Dropout." While Deep Ensembles are accurate, they are computationally ruinous for large-scale tasks. Conversely, MC Dropout is fast but frequently underestimates uncertainty.

This paper introduces a highly efficient "middle way": using Snapshot Ensembles (SE). By capturing multiple model states from a single training run, SE identifies informative data with high accuracy. The result is a simple algorithm that outperforms traditional ensembles while being up to 26x faster.

Proposed Methods

The authors propose two main innovations to overhaul the active learning pipeline:

1. AL with Snapshot Ensembles (SE) Instead of training five entirely different models from scratch to form a Deep Ensemble, the authors train just one model. During the final stages of the optimization path, they save "snapshots" (checkpoints) at regular intervals. These snapshots act as a committee to calculate standard acquisition functions like BALD (Bayesian Active Learning by Disagreement) or Maximum Entropy.

2. SE + Fine-Tuning (FT) Standard AL is highly wasteful; it usually throws away the model and retrains it from scratch every time a new batch of labels is acquired. The authors propose maintaining a single learning trajectory throughout all active learning episodes.

Learning trajectories of SE+FT compared to Vanilla AL with SE.

To prevent the model from "forgetting" older data or overfitting to the small, newly labeled batches, they utilize a replay buffer and $\ell_2$ regularization—techniques heavily inspired by continual learning.

Key Insights

  • Diversity Over Raw Accuracy: A surprising finding is that while individual snapshots in an SE might be slightly less accurate than members of a Deep Ensemble, their predictions on difficult samples are more diverse. For the specific task of selecting data, this disagreement is actually more valuable than raw individual accuracy.
  • Wandering the Optima: The success of SE is rooted in "mode connectivity." By maintaining a high learning rate during the snapshot phase, the model "wanders" around a wide area of the loss surface. This allows it to capture fundamentally different "views" of the data's difficulty.
  • Efficiency is Free: The Fine-Tuning (FT) approach doesn't just save compute time; it maintains a more stable acquisition process. Snapshots gathered during a continuous trajectory prove better at identifying truly difficult samples compared to models started from different random seeds.

Key Results

The researchers evaluated their approach on standard image classification benchmarks, including CIFAR-10, CIFAR-100, and Tiny ImageNet (using ResNet-18, ResNet-50, and Vision Transformer backbones).

  • Massive Speedups: The efficiency gains are dramatic. On CIFAR-10, the proposed SE + FT method took only 2.3 hours to reach peak accuracy. In stark contrast, a standard Deep Ensemble (DE) approach took 60.5 hours—a roughly 26x reduction in compute time.
Performance and runtime comparison on CIFAR-10.
  • Superior Performance: Despite the massive reduction in compute, SE-based active learning consistently achieved higher test accuracy than Deep Ensembles and MC Dropout across almost all tested acquisition functions (VR, BALD, ME, MAR) on both CIFAR-10 and CIFAR-100.
  • Pretrained Power: When fine-tuning pretrained Vision Transformers (ViT) on Tiny ImageNet, SE significantly outperformed Deep Ensembles, particularly in the earliest stages when the number of labeled samples was very small.

Limitations and Future Work

While highly effective, the approach does have a few limitations:

  • Domain Specificity: The paper focuses exclusively on image classification. It remains to be seen if these snapshot-based uncertainty gains translate as effectively to Natural Language Processing (NLP) or complex regression tasks.
  • Hyperparameter Tuning: The "snapshot" phase requires careful tuning of the learning rate and the "burn-in" time. If snapshots are taken too early before the model converges, the quality of data selection drops significantly.
  • Replay Buffer Complexity: The fine-tuning method relies on a replay buffer. Managing the size and composition of this buffer adds a layer of complexity to the training pipeline that standard retraining avoids.

Summary updated

Abstract

Given an unlabeled pool of data and the experts who can label them, active learning aims to build an agent that can effectively acquire data to be queried to the experts, maximizing the gain in performance when trained with them. While there are several principles for active learning, a prevailing approach is to estimate uncertainties of predictions for unlabeled samples and use them to define acquisition functions. Active learning with the uncertainty principle works well for deep learning, especially for large-scale image classification tasks with deep neural networks. Still, it is often overlooked how the uncertainty of predictions is estimated, despite the common findings on the difficulty of accurately estimating uncertainties of deep neural networks. In this paper, we highlight the effectiveness of snapshot ensembles for deep active learning. Compared to the previous approaches based on Monte-Carlo dropout or deep ensembles, we show that a simple acquisition strategy based on uncertainties estimated from parameter snapshots gathered from a single optimization path significantly improves the quality of the acquired samples. Based on this observation, we further propose an efficient active learning algorithm that maintains a single learning trajectory throughout the entire active learning episodes, unlike the existing algorithms training models from scratch for every active learning episode. Through the extensive empirical comparison, we demonstrate the effectiveness of snapshot ensembles for deep active learning.
BibTeX
@inproceedings{jung2023simple,
  author = {Seohyeon Jung and Sanghyun Kim and Juho Lee},
  title = {{A Simple Yet Powerful Deep Active Learning With Snapshots Ensembles}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=IVESH65r0Ar},
}
Figure