Extracurricular Learning: Knowledge Transfer Beyond Empirical Distribution

Hadi Pouransari, Oncel Tuzel

Abstract

Knowledge distillation has been used to transfer knowledge learned by a sophisticated model (teacher) to a simpler model (student). This technique is widely used to compress model complexity. However, in most applications the compressed student model suffers from an accuracy gap with its teacher. We propose extracurricular learning, a novel knowledge distillation method, that bridges this gap by (1) modeling student and teacher output distributions; (2) sampling examples from an approximation to the underlying data distribution; and (3) matching student and teacher output distributions over this extended set including uncertain samples. We conduct rigorous evaluations on regression and classification tasks and show that compared to the standard knowledge distillation, extracurricular learning reduces the gap by 46% to 68%. This leads to major accuracy improvements compared to the empirical risk minimization-based training for various recent neural network architectures: 16% regression error reduction on the MPIIGaze dataset, +3.4% to +9.1% improvement in top-1 classification accuracy on the CIFAR100 dataset, and +2.9% top-1 improvement on the ImageNet dataset.

Introduction

Knowledge Distillation (KD) is a cornerstone technique for deploying high-performance neural networks on resource-constrained hardware. By training a compact "student" model to mimic a larger "teacher" model, practitioners aim to retain as much accuracy as possible while reducing computational overhead. Despite its popularity, a persistent accuracy gap remains between the teacher and the student.

This research identifies two primary bottlenecks in standard KD: the reliance on limited empirical training sets and the neglect of teacher uncertainty. On standard training data, a teacher is often overconfident, providing little "dark knowledge" for the student to learn from. Furthermore, treating teacher outputs as fixed targets ignores the inherent uncertainty the teacher may have about complex or ambiguous samples.

The Extracurricular Learning (XCL) Framework

To address these issues, we introduce Extracurricular Learning (XCL). This framework expands the training set beyond the original empirical distribution to include "extracurricular" samples—unlabeled data points where the teacher’s uncertainty is most informative.

Uncertainty and Data Modeling

XCL explicitly models the teacher’s output as a distribution rather than a point estimate. In regression tasks, outputs are modeled as Gaussians to capture heteroscedastic uncertainty. In classification, the categorical distribution is used. The student is then trained to minimize the KL-divergence between its own output distribution and that of the teacher.

To generate informative samples, XCL approximates the underlying data manifold using two primary methods:

  1. XCL-Mix: New samples are created by blending image pairs in pixel space. Unlike standard MixUp, which assumes linear label transitions, XCL uses the teacher as a non-linear labeler.
  2. XCL-GAN: A generative model (e.g., BigGAN) is used to sample from the data manifold, specifically targeting ambiguous regions by interpolating between class vectors.
XCL-Mix visualization: The teacher model acts as a non-linear interpolator, identifying intermediate classes like 'yurt' when mixing 'sandbar' and 'volcano'.

As shown in the visualization above, the teacher provides nuanced reasoning for mixed samples. A blend of 'sandbar' and 'volcano' might be identified as a 'yurt' by the teacher; XCL ensures the student learns these complex decision boundaries.

Experimental Results

We evaluated XCL across multiple benchmarks, including MPIIGaze (regression), CIFAR100, and ImageNet (classification). XCL consistently outperformed standard KD and state-of-the-art augmentation techniques like MixUp. Across various architectures, XCL reduced the teacher-student accuracy gap by 46% to 68%.

Performance Gains and Data Efficiency

Notable improvements include a 16% reduction in regression error on MPIIGaze and top-1 accuracy boosts of up to +9.1% on CIFAR100. One of the most significant findings is XCL's robustness in low-data regimes.

Test error vs. dataset size: XCL-Mix maintains high performance even with significantly reduced training data compared to standard KD and MixUp.

As illustrated in the figure above, XCL-Mix maintains a much lower test error as the available training data decreases, demonstrating that the "extracurricular" samples effectively compensate for a lack of labeled empirical data.

Conclusion

Extracurricular Learning demonstrates that the performance gap in knowledge distillation is largely due to a lack of challenging, off-manifold examples during training. By modeling uncertainty and leveraging the teacher as a non-linear interpolator for synthetic data, XCL provides a powerful framework for model compression. While generating these extra samples adds to training time, the resulting student models offer significantly higher accuracy with no additional inference cost.

Summary updated

Abstract

Knowledge distillation has been used to transfer knowledge learned by a sophisticated model (teacher) to a simpler model (student). This technique is widely used to compress model complexity. However, in most applications the compressed student model suffers from an accuracy gap with its teacher. We propose extracurricular learning, a novel knowledge distillation method, that bridges this gap by (1) modeling student and teacher output distributions; (2) sampling examples from an approximation to the underlying data distribution; and (3) matching student and teacher output distributions over this extended set including uncertain samples. We conduct rigorous evaluations on regression and classification tasks and show that compared to the standard knowledge distillation, extracurricular learning reduces the gap by 46% to 68%. This leads to major accuracy improvements compared to the empirical risk minimization-based training for various recent neural network architectures: 16% regression error reduction on the MPIIGaze dataset, +3.4% to +9.1% improvement in top-1 classification accuracy on the CIFAR100 dataset, and +2.9% top-1 improvement on the ImageNet dataset.
BibTeX
@inproceedings{pouransari2021extracurricular,
  author = {Hadi Pouransari and Oncel Tuzel},
  title = {{Extracurricular Learning: Knowledge Transfer Beyond Empirical Distribution}},
  booktitle = {Workshops @ CVPR},
  year = {2021},
  url = {https://openreview.net/forum?id=oPeZIqlq7K},
}
Figure