Diversity Matters When Learning From Ensembles

Giung Nam, Jongmin Yoon, Yoonho Lee, Juho Lee

Abstract

Deep ensembles are widely regarded as the gold standard for model uncertainty and prediction accuracy in large-scale image classification. However, their significant computational and memory requirements often preclude deployment in resource-constrained environments. While knowledge distillation (KD) aims to compress an ensemble into a single student model, a persistent performance gap remains. This paper identifies a primary cause: standard KD fails to capture the functional diversity of the ensemble because members often produce nearly identical outputs on training data. We propose Output Diversified Sampling (ODS), a perturbation strategy that forces ensemble members to reveal their differences by seeking inputs where their outputs disagree. Our results demonstrate that ODS-distilled models effectively absorb ensemble diversity, significantly closing the performance gap and improving calibration under distribution shift.

The Training Set Blind Spot

Deep Ensembles (DE) achieve superior performance by training multiple models with different initializations. This process creates a set of functions that agree on the training data but vary in their generalizations. In modern deep learning, ensemble members typically achieve near-zero training error, meaning their outputs on the training set are virtually indistinguishable.

Figure 1: Examples of training images and their corresponding high-confidence, low-diversity prediction distributions. On such samples, standard distillation provides no information about the functional differences between ensemble members.

Because standard knowledge distillation relies on these training samples, the student model is "blind" to the very diversity that makes the ensemble effective. If the teachers do not exhibit disagreement during training, the student cannot learn the nuances of the ensemble's collective decision boundaries.

Methodology: Output Diversified Sampling (ODS)

To overcome this limitation, we introduce a framework that actively reveals the ensemble's internal diversity during the distillation process.

1. Efficient Student Architecture

We utilize BatchEnsemble as the student model. BatchEnsemble is a memory-efficient architecture that uses rank-one factors to simulate multiple subnetworks within a single model. Each subnetwork is tasked with mimicking a specific member of the teacher ensemble.

2. Diversifying the Input Space

The core of our approach is Output Diversified Sampling (ODS). Instead of training solely on original images ($x$), we generate perturbed versions ($\tilde{x}$) by moving in a direction that maximizes the change in the model's output. This is achieved through a stochastic approximation: we randomly select one teacher, calculate the gradient of its output with respect to the input, and apply this perturbation to the image used for training all student subnetworks.

3. Confidence-Scaled Perturbations (ConfODS)

We further refine this with ConfODS, where the perturbation magnitude is scaled by the teacher's confidence. When a teacher is highly confident, the algorithm applies a larger perturbation to find the "boundaries" where ensemble members are likely to diverge, thereby exposing more functional diversity to the student.

Experimental Results

We evaluated our method across several benchmarks, including CIFAR-10/100, TinyImageNet, and corrupted datasets (CIFAR-100-C) to test calibration.

  • Closing the Accuracy Gap: On CIFAR-100, a BatchEnsemble-4 student distilled with ODS achieved 81.88% accuracy, compared to 80.51% with vanilla distillation, nearing the full teacher ensemble's 82.52%.
  • Deep Ensemble Equivalent (DEE): This metric quantifies the student's performance relative to the number of full models in an ensemble. ODS-distilled models achieved a DEE of 3.33, indicating that a single efficient model can match the performance of more than three full-sized models.
  • Robustness and Calibration: ODS significantly improved the Expected Calibration Error (ECE) on corrupted data. On CIFAR-100-C, ODS reduced ECE from 0.216 (vanilla KD) to 0.104, nearly matching the teacher ensemble's 0.096.

Discussion and Insights

Our analysis reveals that ODS is effectively performing Approximate Jacobian Matching. By perturbing the inputs, the student learns not just the teacher's output values, but the local geometry (the "shape") of the teacher's function.

Crucially, we found that perturbations calculated for one teacher are highly transferable to others. This "Jacobian transferability" allows ODS to remain computationally efficient, requiring only one additional gradient calculation per training step. Unlike random Gaussian noise, which often pushes data into meaningless regions of the input space, ODS leverages the model's own gradients to explore meaningful functional variations.

Conclusion

Capturing ensemble diversity is critical for successful knowledge distillation. By using Output Diversified Sampling to expose the hidden disagreements between ensemble members, we can train highly efficient student models that retain the accuracy and uncertainty benefits of large-scale deep ensembles. This approach provides a practical path for deploying state-of-the-art ensemble performance in real-world, resource-constrained applications.

Summary updated

Abstract

Deep ensembles excel in large-scale image classification tasks both in terms of prediction accuracy and calibration. Despite being simple to train, the computation and memory cost of deep ensembles limits their practicability. While some recent works propose to distill an ensemble model into a single model to reduce such costs, there is still a performance gap between the ensemble and distilled models. We propose a simple approach for reducing this gap, i.e., making the distilled performance close to the full ensemble. Our key assumption is that a distilled model should absorb as much function diversity inside the ensemble as possible. We first empirically show that the typical distillation procedure does not effectively transfer such diversity, especially for complex models that achieve near-zero training error. To fix this, we propose a perturbation strategy for distillation that reveals diversity by seeking inputs for which ensemble member outputs disagree. We empirically show that a model distilled with such perturbed samples indeed exhibits enhanced diversity, leading to improved performance.
BibTeX
@inproceedings{nam2021diversity,
  author = {Giung Nam and Jongmin Yoon and Yoonho Lee and Juho Lee},
  title = {{Diversity Matters When Learning From Ensembles}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=f_eOQN87eXc},
}
Figure