Introduction
Deep Ensembles (DE)—training multiple models and averaging their outputs—are the gold standard for achieving high accuracy and reliable uncertainty estimation in deep learning. However, their heavy computational cost hinders their application in resource-limited environments, as they require $M$ times more memory and time during inference.
To address this, researchers often use Knowledge Distillation (KD) to "teach" a single student model to mimic the ensemble. The drawback is that standard KD typically targets the mean prediction of the ensemble. This washes out the functional diversity—the nuanced disagreements between individual models—that provides an ensemble with its robustness. While some parameter-efficient methods like BatchEnsemble use "subnetworks" to retain this diversity, they still require multiple forward passes, defeating the goal of a truly lightweight student.
This paper introduces LatentBE, a method that trains multiple subnetworks but averages their weights for inference, alongside a novel perturbation strategy that forces the student to learn specifically where ensemble members disagree. The result is a single model with the inference speed of one network but the accuracy and uncertainty calibration of a full ensemble.
Methods
The authors propose two primary innovations to bridge the gap between ensemble performance and single-model efficiency:
1. LatentBE (Weight-Averaged Student)
Instead of a standard single network, the authors use a student with multiple "subnetworks" based on BatchEnsemble (BE). In standard BE, subnetworks share most weights but have unique, low-rank "rank-one factors." Normally, you would run all subnetworks and average their outputs.
In LatentBE, the authors train these subnetworks to stay within the same "low-loss region" of the parameter space. After training, they simply average the weights of these rank-one factors to create a single, standard neural network.
2. Diversifying Perturbation (TDiv-SDiv)
To ensure the student actually learns the ensemble's diversity, the authors introduce a new data perturbation strategy. During training, they slightly alter input images to find "weak points" where:
- The Teachers Disagree (High Teacher Diversity or TDiv).
- The Student Subnetworks Agree (Low Student Diversity or SDiv).
By training on these perturbed images, the student is forced to "stretch" its internal diversity to match the complex decision boundaries of the teachers.
Key Insights and Ideas
- The "Flat Minima" Advantage: The success of LatentBE hinges on the subnetworks occupying a single, broad "valley" in the loss landscape. As illustrated in the schematic above, LatentBE eliminates the high-error "loss barrier" between subnetworks that exists in standard BatchEnsemble. This shared low-loss region makes weight averaging possible without destroying performance.
- Closing the Diversity Gap: Previous perturbation methods only looked at what the teachers were doing. By looking at the difference between teacher and student diversity (TDiv - SDiv), the authors target the specific areas where the student is failing to capture the ensemble's complexity.
- Ensemble Performance, Single-Model Cost: The most significant insight is that you do not need to keep the ensemble structure at inference time to reap its benefits. Proper training and weight averaging can effectively "bake" that diversity into a single set of weights.
Key Results
The authors evaluated their method across several standard image classification benchmarks, including CIFAR-10/100, TinyImageNet, and ImageNet-1k, as well as corrupted and Out-of-Distribution (OOD) datasets.
- Accuracy Boost: On CIFAR-100, LatentBE with the TDiv-SDiv perturbation achieved 80.02% accuracy, significantly higher than standard KD (79.09%) and even outperforming the original BatchEnsemble (78.92%), which is much slower.
- Superior Calibration: The method drastically improved uncertainty estimation. On CIFAR-100, the Negative Log-Likelihood (NLL) dropped from 1.038 (KD) to 0.792 (Ours).
- OOD Detection: The model is much better at signaling when it is looking at something it wasn't trained on. It produces much higher entropy (uncertainty) on out-of-distribution data compared to other distillation methods, behaving more like a true ensemble.
- Inference Efficiency: LatentBE maintains an inference time of 0.107 ms/img, whereas a standard 4-model ensemble takes 0.390 ms/img.
Limitations and Future Work
- Training Cost: While inference is fast, training is more expensive because it requires extra forward and backward passes through the teacher ensemble to compute the diversifying perturbations.
- Architecture Constraints: The current implementation relies on the specific rank-one structure of BatchEnsemble. It remains to be seen how well this generalizes to other types of parameter-efficient ensembles.
- Hyperparameter Sensitivity: The method introduces new hyperparameters, such as the perturbation step size and weight decay for rank-one factors, which may require careful tuning for new architectures or datasets.
Summary updated