Distilling Ensembles Improves Uncertainty Estimates

Zelda E Mariet, Rodolphe Jenatton, Florian Wenzel, Dustin TranAABI 2020 · OpenReview

TL;DR

This paper tackles the challenge of achieving reliable uncertainty estimates from deep learning models, a crucial aspect for real-world applications. The authors demonstrate that "distilling" the knowledge from computationally expensive deep ensembles into more efficient "batch ensembles" significantly improves uncertainty estimation without sacrificing accuracy, and even outperforms standard batch ensembles.

What Problem Does This Paper Address?

Deep neural networks are increasingly used for critical tasks like medical diagnosis and weather forecasting, where not only accurate predictions but also reliable estimates of uncertainty are essential. While deep ensembles (multiple independently trained deep networks) offer good accuracy and uncertainty estimates, they are computationally expensive. Batch ensembles (ensembles of networks sharing parameters) offer a more efficient alternative but historically haven't matched the performance of deep ensembles, especially in terms of uncertainty. This paper aims to bridge this performance gap by exploring how to transfer the knowledge from deep ensembles to batch ensembles, making robust uncertainty estimation more accessible.

Methods

The authors first investigate whether the weights of a deep ensemble can be directly approximated by the weights of a batch ensemble. They present theoretical results suggesting this is unlikely, indicating that batch ensembles don't simply offer a compressed representation of deep ensembles.

Given this, they turn to knowledge distillation. This technique involves training a "student" model (a batch ensemble in this case) to mimic the output of a "teacher" model (a deep ensemble). Specifically, they explore two distillation strategies:

  • Knowledge Distillation: The student ensemble predicts the average output of the teacher ensemble. This is more flexible as it doesn't require the student and teacher to have the same number of members.
  • Head Distillation: Each member of the student ensemble is trained to mimic a specific member of the teacher ensemble. The authors hypothesize this might preserve diversity but find it less effective.

They also explore initializing batch ensembles with weights from deep ensembles, but find this approach performs poorly.

Datasets

The experiments were conducted on the following datasets:

  • Cifar-10
  • Cifar-100

These are standard image classification datasets commonly used for evaluating deep learning models.

Key Insights and Ideas

  • Batch ensembles are not just compressed deep ensembles: Theoretical analysis shows that the weight structure of batch ensembles is fundamentally different from deep ensembles, meaning direct weight approximation is not a viable path to matching performance.
  • Distillation is the key: Transferring the functional knowledge (predictions) from deep ensembles to batch ensembles via distillation is highly effective.
  • Knowledge distillation outperforms head distillation: The more flexible approach of having the student ensemble learn the teacher's average prediction is more successful than trying to match individual ensemble members.
  • Distillation improves uncertainty, not just accuracy: The primary benefit observed is a significant improvement in calibration metrics (like ECE and NLL), bringing batch ensembles closer to the performance of deep ensembles.
  • Single distilled networks fall short: Distilling a deep ensemble into a single network, even one with a similar parameter count to a batch ensemble, does not yield comparable improvements in uncertainty estimation.

Key Results

  • Improved Uncertainty: Distilled batch ensembles significantly improve calibration metrics (ECE and NLL) on both standard and corrupted test data, matching or even exceeding the performance of the teacher deep ensembles.
  • Comparable Accuracy: While the focus is on uncertainty, distilled batch ensembles maintain competitive accuracy, often matching or slightly improving upon standard batch ensembles.
  • Efficiency Gains: Distilled batch ensembles achieve these improvements with a substantial reduction in memory footprint (e.g., 4x decrease) compared to deep ensembles.
  • Initialization Fails: Initializing batch ensembles with deep ensemble weights leads to poor performance, highlighting the importance of the distillation process itself.

Limitations and Future Work

  • Diversity Inheritance: Preliminary results suggest that the diversity of ensemble members might not be naively inherited through distillation. The authors leave open the question of the exact mechanism behind distillation's success.
  • Hyperparameter Diversity: The paper notes that hyperparameter diversity, which can be beneficial, might not be easily inherited by batch ensembles.
  • Future Directions: The authors suggest exploring how knowledge distillation can be used in a more modular way, allowing for independent changes in teacher and student architectures. They also point to the need to understand the underlying cause of distillation's success, particularly regarding diversity.

Summary updated

Abstract

We seek to bridge the performance gap between batch ensembles (ensembles of deep networks with shared parameters) and deep ensembles on tasks which require not only predictions, but also uncertainty estimates for these predictions. We obtain negative theoretical results on the possibility of approximating deep ensemble weights by batch ensemble weights, and so turn to distillation. Training a batch ensemble on the outputs of deep ensembles improves accuracy and uncertainty estimates, without requiring hyper-parameter tuning. This result is specific to the choice of batch ensemble architectures: distilling deep ensembles to a single network is unsuccessful, despite single networks having only marginally fewer parameters than batch ensembles.
BibTeX
@inproceedings{mariet2020distilling,
  author = {Zelda E Mariet and Rodolphe Jenatton and Florian Wenzel and Dustin Tran},
  title = {{Distilling Ensembles Improves Uncertainty Estimates}},
  booktitle = {AABI},
  year = {2020},
  url = {https://openreview.net/forum?id=Lzi5IMyJTFX},
}
Figure