Improving Ensemble Distillation With Weight Averaging and Diversifying Perturbation

Giung Nam, Hyungi Lee, Byeongho Heo, Juho Lee

Abstract

Ensembles of deep neural networks have demonstrated superior performance, but their heavy computational cost hinders applying them for resource-limited environments. It motivates distilling knowledge from the ensemble teacher into a smaller student network, and there are two important design choices for this ensemble distillation: 1) how to construct the student network, and 2) what data should be shown during training. In this paper, we propose a weight averaging technique where a student with multiple subnetworks is trained to absorb the functional diversity of ensemble teachers, but then those subnetworks are properly averaged for inference, giving a single student network with no additional inference cost. We also propose a perturbation strategy that seeks inputs from which the diversities of teachers can be better transferred to the student. Combining these two, our method significantly improves upon previous methods on various image classification tasks.

Introduction

Deep Ensembles (DE)—the practice of training multiple versions of the same model and averaging their predictions—are widely considered the gold standard for predictive accuracy and uncertainty estimation. However, their deployment is often prohibited in resource-constrained environments because running an ensemble of $N$ models requires $N$ times the memory and computation.

Knowledge Distillation (KD) is the standard solution to this problem, where a single "student" model is trained to mimic the ensemble "teacher." However, standard KD often fails to capture the functional diversity of the ensemble. When the outputs of ensemble members are averaged to create a target for the student, the unique nuances and disagreements that make the ensemble robust are often "washed out." This paper addresses this gap by rethinking both the student architecture and the data used for training.

Methods

The authors propose two primary innovations to bridge the performance gap between ensembles and distilled students without increasing inference costs.

1. LatentBE: Weight-Averaged Student

To capture the diversity of the teacher ensemble, the authors utilize the BatchEnsemble architecture as a starting point. In BatchEnsemble, a model uses shared weights combined with small, rank-one multiplicative factors to create distinct "subnetworks."

While previous methods kept these subnetworks separate during inference (maintaining a computational overhead), the authors introduce LatentBE. During training, these subnetworks are distilled one-to-one from individual ensemble members. Crucially, the training process ensures these subnetworks stay within the same "low-loss region" of the weight space. After training, the weights of these subnetworks are averaged into a single set of parameters. This results in a single model that has absorbed the functional diversity of the ensemble but operates at the speed of a standard network with zero additional inference cost.

2. Diversifying Perturbation

The authors argue that standard training data is often insufficient for transferring diversity. If all ensemble members agree on a specific training point, there is no "disagreement" for the student to learn from.

To solve this, they propose a Diversifying Perturbation strategy. This method modifies input images during training using gradients to find "weak points" where:

  • Teacher Diversity is high: The ensemble members disagree on the output.
  • Student Diversity is low: The student's internal subnetworks (prior to averaging) are in consensus.

By training on these specifically perturbed inputs, the student is forced to learn the exact boundaries where the ensemble members differ, leading to significantly better generalization and uncertainty awareness.

Experimental Results

The proposed method was evaluated on several standard benchmarks, including CIFAR-10/100, TinyImageNet, and ImageNet-1k, as well as robustness benchmarks like CIFAR-10-C.

Accuracy and Calibration

On CIFAR-100, the combination of LatentBE and Diversifying Perturbation achieved 80.02% accuracy, a notable improvement over vanilla KD (79.09%). Beyond raw accuracy, the method drastically reduced the Expected Calibration Error (ECE). This indicates that the model is not only more accurate but also more "honest" about its confidence levels, avoiding the overconfidence common in standard neural networks.

Robustness and OOD Detection

The method demonstrated superior performance on corrupted datasets (CIFAR-C), maintaining higher accuracy and lower negative log-likelihood than competitive baselines. Furthermore, in Out-of-Distribution (OOD) detection tasks using datasets like SVHN and LSUN, the model showed higher entropy on unseen data, making it safer for real-world deployment where encountering unexpected inputs is common.

Conclusion

By combining weight-averaged subnetworks with a targeted perturbation strategy, this research provides a highly efficient way to distill the benefits of large ensembles into single-model architectures. While the training process is more computationally intensive due to the perturbation steps, the resulting student model offers state-of-the-art performance with no additional cost at test time. Future work may explore the application of this "train-multiple-average-later" approach to other parameter-efficient architectures beyond BatchEnsemble.

Summary updated

Abstract

Ensembles of deep neural networks have demonstrated superior performance, but their heavy computational cost hinders applying them for resource-limited environments. It motivates distilling knowledge from the ensemble teacher into a smaller student network, and there are two important design choices for this ensemble distillation: 1) how to construct the student network, and 2) what data should be shown during training. In this paper, we propose a weight averaging technique where a student with multiple subnetworks is trained to absorb the functional diversity of ensemble teachers, but then those subnetworks are properly averaged for inference, giving a single student network with no additional inference cost. We also propose a perturbation strategy that seeks inputs from which the diversities of teachers can be better transferred to the student. Combining these two, our method significantly improves upon previous methods on various image classification tasks.
BibTeX
@inproceedings{nam2022improving,
  author = {Giung Nam and Hyungi Lee and Byeongho Heo and Juho Lee},
  title = {{Improving Ensemble Distillation With Weight Averaging and Diversifying Perturbation}},
  booktitle = {International Conference on Machine Learning},
  year = {2022},
  url = {https://openreview.net/forum?id=t5bB98aGfRH},
}
Figure