On the Orthogonality of Knowledge Distillation with Other Techniques: From an Ensemble Perspective

Seonguk Park, KiYoon Yoo, Nojun KwakarXiv 2020 · arXiv

Introduction

As deep neural networks (DNNs) scale in complexity, the demand for efficient deployment has led to a diverse toolbox of optimization techniques. These include pruning (removing redundant weights), quantization (reducing precision), data augmentation (e.g., CutMix, AutoAugment), and knowledge distillation (KD) (mimicking a teacher model). While these methods are often used in isolation, a critical question remains: are they redundant, or do they offer additive benefits?

This research explores the concept of orthogonality—the idea that these techniques operate through distinct mechanisms and can be combined for cumulative performance gains. The authors argue that KD is the "universal donor" of model efficiency, consistently boosting accuracy even when models are already heavily optimized.

Knowledge Distillation as Implicit Ensembling

The paper re-interprets KD through an ensemble lens, specifically comparing it to bagging (bootstrap aggregating). Unlike standard training that relies on one-hot "hard" labels, KD utilizes the teacher's soft probability distributions. This effectively increases the "effective number of samples" and allows the student to learn from a weighted average of targets. This mechanism mimics an ensemble effect without the prohibitive inference-time cost of running multiple models, explaining why even "Born Again" students (same architecture as the teacher) can outperform their mentors.

Synergy with Structured Pruning

To test orthogonality, the authors paired KD with structured pruning methods like L1-norm filter pruning. They introduced a "Surplus" metric to quantify synergy: the gain achieved by the combination that exceeds the sum of individual improvements.

Performance gains and 'Surplus' metrics for ResNet architectures combining pruning and distillation.

As shown in the experimental results for ResNet-56 and ResNet-110, combining pruning with KD consistently yielded positive surplus values. For instance, on ResNet-56 with 50-70% remaining channels, the combination resulted in a surplus gain of 1.50%. This indicates that KD does more than just recover accuracy lost during pruning; it provides an independent performance boost that pruning cannot access.

Synergy with Data Augmentation

The study also found powerful interactions between KD and input-level regularizations like AutoAugment and CutMix. Because these techniques augment different stages of the training pipeline—inputs versus outputs—they are naturally compatible. On PyramidNet, combining AutoAugment with KD reached an accuracy of 87.27%, a total boost of 3.81% over the baseline, which significantly outperformed the individual contributions of each method (1.76% and 0.50% respectively).

Qualitative Analysis: Loss Landscape Smoothing

Beyond accuracy metrics, the authors investigated how KD physically alters the optimization process. Using loss landscape visualizations, they compared models trained from scratch, with CutMix, and with Distillation.

Loss landscape visualizations comparing Scratch, CutMix, and Distillation training across decay steps.

While standard training and CutMix often result in sharp, narrow local minima—which are prone to poor generalization—KD draws these minima together into a visibly smoother and wider basin. This smoothing effect facilitates better optimization and explains the robust generalization observed in distilled models.

Conclusion

The findings confirm that Knowledge Distillation is fundamentally orthogonal to pruning and data augmentation. By acting as an implicit ensemble and smoothing the loss landscape, KD provides unique benefits that complement other efficiency-enhancing techniques. For practitioners, this suggests that KD should be a standard component of any model compression pipeline, as it consistently offers "free" performance gains regardless of other optimizations employed.

Summary updated

Abstract

To put a state-of-the-art neural network to practical use, it is necessary to design a model that has a good trade-off between the resource consumption and performance on the test set. Many researchers and engineers are developing methods that enable training or designing a model more efficiently. Developing an efficient model includes several strategies such as network architecture search, pruning, quantization, knowledge distillation, utilizing cheap convolution, regularization, and also includes any craft that leads to a better performance-resource trade-off. When combining these technologies together, it would be ideal if one source of performance improvement does not conflict with others. We call this property as the orthogonality in model efficiency. In this paper, we focus on knowledge distillation and demonstrate that knowledge distillation methods are orthogonal to other efficiency-enhancing methods both analytically and empirically. Analytically, we claim that knowledge distillation functions analogous to a ensemble method, bootstrap aggregating. This analytical explanation is provided from the perspective of implicit data augmentation property of knowledge distillation. Empirically, we verify knowledge distillation as a powerful apparatus for practical deployment of efficient neural network, and also introduce ways to integrate it with other methods effectively.
BibTeX
@misc{park2020orthogonality,
  author = {Seonguk Park and KiYoon Yoo and Nojun Kwak},
  title = {{On the Orthogonality of Knowledge Distillation with Other Techniques: From an Ensemble Perspective}},
  year = {2020},
  eprint = {2009.04120},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2009.04120},
}
Figure