Deep ensembles are widely considered the "gold standard" for predictive performance and uncertainty estimation in deep learning. By training the same architecture multiple times with different initializations, practitioners create a committee of models that explore different modes of the loss landscape. However, deploying an ensemble of five or ten large models at inference time is computationally expensive and memory-intensive.
The standard solution to this bottleneck is Knowledge Distillation (KD)—training a single, smaller "student" model to mimic the predictions of the "teacher" ensemble. But there is a catch: distilled models rarely match the performance of the full ensemble. A NeurIPS 2021 paper from researchers at KAIST and Stanford argues that this performance gap stems from a lack of diversity during the distillation process.
By using Output Diversified Sampling (ODS) to perturb training inputs into regions where ensemble members disagree, the authors successfully transfer ensemble diversity to a student model, significantly closing the performance gap.
The Problem: Hidden Diversity on Training Data
The effectiveness of Deep Ensembles stems from the fact that different members make different mistakes. When their predictions are averaged, these individual errors cancel out, leading to better overall accuracy and more reliable uncertainty estimates.
However, if you evaluate a fully trained ensemble on its own training set, every member is usually "correct" and highly confident. Their outputs are virtually indistinguishable.
If a student model only observes the ensemble's behavior on these familiar training points, it learns that all ensemble members are identical. It fails to capture the diverse "functional behavior" that exists just slightly off the training manifold—the exact diversity that makes ensembles robust to new, unseen data. Standard KD is akin to trying to learn a teacher's personality by only watching them sleep; it misses how they react differently to stress or novel situations.
The Solution: One-to-One Distillation with ODS
To reveal this hidden diversity, the authors propose a simple but effective fix using Output Diversified Sampling (ODS).
- The Student Architecture: The researchers utilize BatchEnsemble, a lightweight ensemble method where members share most weights but have unique rank-one factors. This allows the student to have multiple subnetworks (or "heads") to match the multiple teachers without the parameter overhead of a full ensemble.
- Output Diversified Sampling (ODS): Instead of distilling on the raw image, the method perturbs it. ODS finds a direction in the input space that maximizes the change in the model's output, effectively "poking" the teachers to see how they react.
- The Training Loop: For every training step, a random teacher is selected. The system calculates an ODS perturbation to find a "disagreement region," and then forces the student subnetworks to match the diverse teacher outputs at that newly perturbed point.
Mathematically, this approach acts as a highly efficient form of Jacobian Matching. Rather than just matching the output values at specific training points, the student learns to match the gradients of the teachers, effectively capturing how the ensemble behaves in the neighborhood of the data. Furthermore, a perturbation calculated for one teacher network often reveals diversity across all teachers, making the method computationally feasible.
Key Results
The researchers evaluated their method on standard image classification benchmarks (CIFAR-10, CIFAR-100, TinyImageNet) and corrupted datasets (CIFAR-10-C) to test distribution shift. The proposed method, particularly the ConfODS variant which scales perturbations by teacher confidence, outperformed all baselines across accuracy and uncertainty metrics.
- Accuracy Boost: On CIFAR-100, distilling a WideResNet-28x10 ensemble into a BatchEnsemble student improved accuracy from 80.51% (standard KD) to 81.88%.
- Uncertainty Calibration: The method significantly reduced the Expected Calibration Error (ECE), leading to more trustworthy confidence scores.
- Deep Ensemble Equivalent (DEE): This metric measures how many "full" models a compressed model is worth. On CIFAR-100, their distilled student achieved a DEE of 3.33, meaning a single lightweight model performed as well as an ensemble of over three full-sized models.
Limitations and Future Work
While highly effective, the approach has limitations. The method still requires training a full Deep Ensemble first to act as the teacher, which is the very computational burden users ultimately want to avoid. Additionally, while the authors provide an interpretation via Jacobian matching, the ODS approach remains somewhat heuristic; a rigorous theoretical framework for "optimal diversity transfer" is still an open challenge. Finally, there is a limit to how much diversity a small student can absorb—if the student network lacks capacity, ODS perturbations might confuse rather than help it.
For researchers and engineers looking to deploy robust models in resource-constrained environments, this paper provides a clear path forward: to capture the true power of an ensemble, don't just distill the labels—distill the diversity.
Summary updated