Extracurricular Learning: Knowledge Transfer Beyond Empirical Distribution

Hadi Pouransari, Mojan Javaheripi, Vinay Sharma, Oncel Tuzel

Abstract

Knowledge distillation has been used to transfer knowledge learned by a sophisticated model (teacher) to a simpler model (student). This technique is widely used to compress model complexity. However, in most applications the compressed student model suffers from an accuracy gap with its teacher. We propose extracurricular learning, a novel knowledge distillation method, that bridges this gap by (1) modeling student and teacher output distributions; (2) sampling examples from an approximation to the underlying data distribution; and (3) matching student and teacher output distributions over this extended set including uncertain samples. We conduct rigorous evaluations on regression and classification tasks and show that compared to the standard knowledge distillation, extracurricular learning reduces the gap by 46% to 68%. This leads to major accuracy improvements compared to the empirical risk minimization-based training for various recent neural network architectures: 16% regression error reduction on the MPIIGaze dataset, +3.4% to +9.1% improvement in top-1 classification accuracy on the CIFAR100 dataset, and +2.9% top-1 improvement on the ImageNet dataset.

Introduction

Knowledge Distillation (KD) is the cornerstone of modern model compression. By training a compact "student" model to mimic the soft outputs of a high-capacity "teacher," we can deploy efficient models that retain much of the teacher's intelligence. However, a persistent performance gap remains: students rarely match their teachers. Extracurricular Learning (XCL) identifies a fundamental flaw in standard KD—overconfidence on the training set—and proposes a framework to bridge this gap by exploring the "uncertain" regions of the data manifold.

The Problem: Overconfidence and the Accuracy Gap

The primary hurdle in distillation is that a well-trained teacher is often too certain on the standard training data (the empirical distribution). When a teacher predicts a class with 99.9% probability, the "dark knowledge"—the subtle relationships between classes contained in the remaining 0.1%—is effectively lost. To learn the teacher's complex decision boundaries, the student needs to see examples where the teacher is uncertain. These informative samples are rarely found in the original training set.

Extracurricular Learning (XCL)

XCL addresses this by moving beyond the training set to find more informative data points. The method rests on three pillars:

1. Modeling Output Distributions

Instead of simple point estimates, XCL models outputs as distributions. For classification, this uses standard categorical distributions. For regression, the authors introduce a novel approach: modeling outputs as Gaussians. By predicting both a mean $\mu$ and a variance $\sigma^2$, the student learns to match the teacher's confidence levels, preventing it from being over-penalized on inherently ambiguous samples.

2. Sampling Uncertain Examples

To find "extracurricular" samples, XCL approximates the underlying data distribution. Two primary methods are proposed:

  • XCL-Mix: New samples are created by blending pairs of images in pixel space. Unlike standard MixUp, XCL uses the teacher's soft labels for these hybrid images, capturing how the teacher's perception shifts between classes.
  • XCL-GAN: A Generative Adversarial Network (e.g., BigGAN) generates images by interpolating between class conditioning vectors, creating visually ambiguous "in-between" images.
Figure 3: Teacher output distribution on interpolated samples between a Pomeranian and a Robin. As the interpolation parameter λ moves from 0 to 1, the teacher identifies intermediate classes like macaques and partridges, revealing the 'dark knowledge' that standard training sets hide.

3. Matching over the Extended Set

The student is trained to minimize the KL divergence between its output distribution and the teacher's across both the original training data and the newly generated extracurricular samples.

Key Results

XCL consistently outperforms standard KD and other state-of-the-art baselines across diverse benchmarks.

  • Gap Reduction: XCL reduces the teacher-student accuracy gap by 46% to 68% compared to standard KD.
  • Classification (CIFAR100 & ImageNet): XCL-Mix provides massive gains, particularly for highly compressed models. For a BinaryNet student on CIFAR100, accuracy improved by +9.1%. On ImageNet, a ResNet-101 student saw a +2.9% top-1 improvement.
  • Data Efficiency: One of the most striking findings is XCL's efficiency. As shown in Figure 4, XCL-Mix trained on a small fraction of the data can outperform standard methods trained on the full dataset.
Figure 4: Test error vs. dataset size on CIFAR100. XCL-Mix significantly outperforms ERM, MixUp, and standard KD, maintaining high performance even when the available training data is severely limited.
  • Regression (MPIIGaze): By modeling uncertainty, XCL achieved a 16% reduction in gaze angle error compared to standard empirical risk minimization (ERM).

Conclusion

Extracurricular Learning demonstrates that the key to better distillation isn't just more data, but the right data. By forcing the student to learn from the teacher's uncertainty on approximated data distributions, XCL unlocks the "dark knowledge" necessary to bridge the accuracy gap. Whether through simple pixel-space mixing or complex GAN-based generation, XCL provides a robust framework for creating compact, high-performing models ready for real-world deployment.

Summary updated

Abstract

Knowledge distillation has been used to transfer knowledge learned by a sophisticated model (teacher) to a simpler model (student). This technique is widely used to compress model complexity. However, in most applications the compressed student model suffers from an accuracy gap with its teacher. We propose extracurricular learning, a novel knowledge distillation method, that bridges this gap by (1) modeling student and teacher output distributions; (2) sampling examples from an approximation to the underlying data distribution; and (3) matching student and teacher output distributions over this extended set including uncertain samples. We conduct rigorous evaluations on regression and classification tasks and show that compared to the standard knowledge distillation, extracurricular learning reduces the gap by 46% to 68%. This leads to major accuracy improvements compared to the empirical risk minimization-based training for various recent neural network architectures: 16% regression error reduction on the MPIIGaze dataset, +3.4% to +9.1% improvement in top-1 classification accuracy on the CIFAR100 dataset, and +2.9% top-1 improvement on the ImageNet dataset.
BibTeX
@inproceedings{pouransari2021extracurricular,
  author = {Hadi Pouransari and Mojan Javaheripi and Vinay Sharma and Oncel Tuzel},
  title = {{Extracurricular Learning: Knowledge Transfer Beyond Empirical Distribution}},
  booktitle = {Workshops @ CVPR},
  year = {2021},
  url = {https://openreview.net/forum?id=oPeZIqlq7K},
}

Loading related content...

Figure