Abstract
Deep ensembles are widely regarded as the gold standard for model uncertainty and prediction accuracy in large-scale image classification. However, their significant computational and memory requirements often preclude deployment in resource-constrained environments. While knowledge distillation (KD) aims to compress an ensemble into a single student model, a persistent performance gap remains. This paper identifies a primary cause: standard KD fails to capture the functional diversity of the ensemble because members often produce nearly identical outputs on training data. We propose Output Diversified Sampling (ODS), a perturbation strategy that forces ensemble members to reveal their differences by seeking inputs where their outputs disagree. Our results demonstrate that ODS-distilled models effectively absorb ensemble diversity, significantly closing the performance gap and improving calibration under distribution shift.
The Training Set Blind Spot
Deep Ensembles (DE) achieve superior performance by training multiple models with different initializations. This process creates a set of functions that agree on the training data but vary in their generalizations. In modern deep learning, ensemble members typically achieve near-zero training error, meaning their outputs on the training set are virtually indistinguishable.
Because standard knowledge distillation relies on these training samples, the student model is "blind" to the very diversity that makes the ensemble effective. If the teachers do not exhibit disagreement during training, the student cannot learn the nuances of the ensemble's collective decision boundaries.
Methodology: Output Diversified Sampling (ODS)
To overcome this limitation, we introduce a framework that actively reveals the ensemble's internal diversity during the distillation process.
1. Efficient Student Architecture
We utilize BatchEnsemble as the student model. BatchEnsemble is a memory-efficient architecture that uses rank-one factors to simulate multiple subnetworks within a single model. Each subnetwork is tasked with mimicking a specific member of the teacher ensemble.
2. Diversifying the Input Space
The core of our approach is Output Diversified Sampling (ODS). Instead of training solely on original images ($x$), we generate perturbed versions ($\tilde{x}$) by moving in a direction that maximizes the change in the model's output. This is achieved through a stochastic approximation: we randomly select one teacher, calculate the gradient of its output with respect to the input, and apply this perturbation to the image used for training all student subnetworks.
3. Confidence-Scaled Perturbations (ConfODS)
We further refine this with ConfODS, where the perturbation magnitude is scaled by the teacher's confidence. When a teacher is highly confident, the algorithm applies a larger perturbation to find the "boundaries" where ensemble members are likely to diverge, thereby exposing more functional diversity to the student.
Experimental Results
We evaluated our method across several benchmarks, including CIFAR-10/100, TinyImageNet, and corrupted datasets (CIFAR-100-C) to test calibration.
- Closing the Accuracy Gap: On CIFAR-100, a BatchEnsemble-4 student distilled with ODS achieved 81.88% accuracy, compared to 80.51% with vanilla distillation, nearing the full teacher ensemble's 82.52%.
- Deep Ensemble Equivalent (DEE): This metric quantifies the student's performance relative to the number of full models in an ensemble. ODS-distilled models achieved a DEE of 3.33, indicating that a single efficient model can match the performance of more than three full-sized models.
- Robustness and Calibration: ODS significantly improved the Expected Calibration Error (ECE) on corrupted data. On CIFAR-100-C, ODS reduced ECE from 0.216 (vanilla KD) to 0.104, nearly matching the teacher ensemble's 0.096.
Discussion and Insights
Our analysis reveals that ODS is effectively performing Approximate Jacobian Matching. By perturbing the inputs, the student learns not just the teacher's output values, but the local geometry (the "shape") of the teacher's function.
Crucially, we found that perturbations calculated for one teacher are highly transferable to others. This "Jacobian transferability" allows ODS to remain computationally efficient, requiring only one additional gradient calculation per training step. Unlike random Gaussian noise, which often pushes data into meaningless regions of the input space, ODS leverages the model's own gradients to explore meaningful functional variations.
Conclusion
Capturing ensemble diversity is critical for successful knowledge distillation. By using Output Diversified Sampling to expose the hidden disagreements between ensemble members, we can train highly efficient student models that retain the accuracy and uncertainty benefits of large-scale deep ensembles. This approach provides a practical path for deploying state-of-the-art ensemble performance in real-world, resource-constrained applications.
Summary updated