Diversity Matters When Learning From Ensembles

Giung Nam, Jongmin Yoon, Yoonho Lee, Juho Lee

Deep ensembles are widely considered the "gold standard" for predictive performance and uncertainty estimation in deep learning. By training the same architecture multiple times with different initializations, practitioners create a committee of models that explore different modes of the loss landscape. However, deploying an ensemble of five or ten large models at inference time is computationally expensive and memory-intensive.

The standard solution to this bottleneck is Knowledge Distillation (KD)—training a single, smaller "student" model to mimic the predictions of the "teacher" ensemble. But there is a catch: distilled models rarely match the performance of the full ensemble. A NeurIPS 2021 paper from researchers at KAIST and Stanford argues that this performance gap stems from a lack of diversity during the distillation process.

By using Output Diversified Sampling (ODS) to perturb training inputs into regions where ensemble members disagree, the authors successfully transfer ensemble diversity to a student model, significantly closing the performance gap.

The Problem: Hidden Diversity on Training Data

The effectiveness of Deep Ensembles stems from the fact that different members make different mistakes. When their predictions are averaged, these individual errors cancel out, leading to better overall accuracy and more reliable uncertainty estimates.

However, if you evaluate a fully trained ensemble on its own training set, every member is usually "correct" and highly confident. Their outputs are virtually indistinguishable.

A model exhibiting high confidence on a training sample, illustrating how ensemble members often produce identical, overconfident outputs on familiar data.

If a student model only observes the ensemble's behavior on these familiar training points, it learns that all ensemble members are identical. It fails to capture the diverse "functional behavior" that exists just slightly off the training manifold—the exact diversity that makes ensembles robust to new, unseen data. Standard KD is akin to trying to learn a teacher's personality by only watching them sleep; it misses how they react differently to stress or novel situations.

The Solution: One-to-One Distillation with ODS

To reveal this hidden diversity, the authors propose a simple but effective fix using Output Diversified Sampling (ODS).

  1. The Student Architecture: The researchers utilize BatchEnsemble, a lightweight ensemble method where members share most weights but have unique rank-one factors. This allows the student to have multiple subnetworks (or "heads") to match the multiple teachers without the parameter overhead of a full ensemble.
  2. Output Diversified Sampling (ODS): Instead of distilling on the raw image, the method perturbs it. ODS finds a direction in the input space that maximizes the change in the model's output, effectively "poking" the teachers to see how they react.
  3. The Training Loop: For every training step, a random teacher is selected. The system calculates an ODS perturbation to find a "disagreement region," and then forces the student subnetworks to match the diverse teacher outputs at that newly perturbed point.

Mathematically, this approach acts as a highly efficient form of Jacobian Matching. Rather than just matching the output values at specific training points, the student learns to match the gradients of the teachers, effectively capturing how the ensemble behaves in the neighborhood of the data. Furthermore, a perturbation calculated for one teacher network often reveals diversity across all teachers, making the method computationally feasible.

Key Results

The researchers evaluated their method on standard image classification benchmarks (CIFAR-10, CIFAR-100, TinyImageNet) and corrupted datasets (CIFAR-10-C) to test distribution shift. The proposed method, particularly the ConfODS variant which scales perturbations by teacher confidence, outperformed all baselines across accuracy and uncertainty metrics.

  • Accuracy Boost: On CIFAR-100, distilling a WideResNet-28x10 ensemble into a BatchEnsemble student improved accuracy from 80.51% (standard KD) to 81.88%.
  • Uncertainty Calibration: The method significantly reduced the Expected Calibration Error (ECE), leading to more trustworthy confidence scores.
  • Deep Ensemble Equivalent (DEE): This metric measures how many "full" models a compressed model is worth. On CIFAR-100, their distilled student achieved a DEE of 3.33, meaning a single lightweight model performed as well as an ensemble of over three full-sized models.

Limitations and Future Work

While highly effective, the approach has limitations. The method still requires training a full Deep Ensemble first to act as the teacher, which is the very computational burden users ultimately want to avoid. Additionally, while the authors provide an interpretation via Jacobian matching, the ODS approach remains somewhat heuristic; a rigorous theoretical framework for "optimal diversity transfer" is still an open challenge. Finally, there is a limit to how much diversity a small student can absorb—if the student network lacks capacity, ODS perturbations might confuse rather than help it.

For researchers and engineers looking to deploy robust models in resource-constrained environments, this paper provides a clear path forward: to capture the true power of an ensemble, don't just distill the labels—distill the diversity.

Summary updated

Abstract

Deep ensembles excel in large-scale image classification tasks both in terms of prediction accuracy and calibration. Despite being simple to train, the computation and memory cost of deep ensembles limits their practicability. While some recent works propose to distill an ensemble model into a single model to reduce such costs, there is still a performance gap between the ensemble and distilled models. We propose a simple approach for reducing this gap, i.e., making the distilled performance close to the full ensemble. Our key assumption is that a distilled model should absorb as much function diversity inside the ensemble as possible. We first empirically show that the typical distillation procedure does not effectively transfer such diversity, especially for complex models that achieve near-zero training error. To fix this, we propose a perturbation strategy for distillation that reveals diversity by seeking inputs for which ensemble member outputs disagree. We empirically show that a model distilled with such perturbed samples indeed exhibits enhanced diversity, leading to improved performance.
BibTeX
@inproceedings{nam2021diversity,
  author = {Giung Nam and Jongmin Yoon and Yoonho Lee and Juho Lee},
  title = {{Diversity Matters When Learning From Ensembles}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=f_eOQN87eXc},
}

Loading related content...

Figure