Progressive Knowledge Distillation: Constructing Ensembles for Efficient Inference

Don Dennis, Abhishek Shetty, Anish Prasad Sevekari, Kazuhito Koishida, Virginia SmithICLR 2023 · OpenReview

Abstract

Knowledge distillation is commonly used to compress an ensemble of models into a single model. In this work we study the problem of progressive distillation: Given a large, pretrained teacher model $g$, we seek to decompose the model into an ensemble of smaller, low-inference cost student models $f_i$. The resulting ensemble allows for flexibly tuning accuracy vs. inference cost, which can be useful for a multitude of applications in efficient inference. Our method, B-DISTIL, uses a boosting procedure that allows function composition based aggregation rules to construct expressive ensembles with similar performance as $g$ using much smaller student models. We demonstrate the effectiveness of B-DISTIL by decomposing pretrained models across a variety of image, speech, and sensor datasets. Our method comes with strong theoretical guarantees in terms of convergence as well as generalization.
BibTeX
@inproceedings{dennis2023progressive,
  author = {Don Dennis and Abhishek Shetty and Anish Prasad Sevekari and Kazuhito Koishida and Virginia Smith},
  title = {{Progressive Knowledge Distillation: Constructing Ensembles for Efficient Inference}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=sE-9hkZL5wV},
}

Loading related content...

Figure