KDGAN: Knowledge Distillation with Generative Adversarial Networks

Xiaojie Wang, Rui Zhang, Yu Sun, Jianzhong Qi

TL;DR

KDGAN is a novel three-player framework that combines Knowledge Distillation (KD) and Generative Adversarial Networks (GANs) to train high-performance, lightweight classifiers. By having a classifier, a teacher, and a discriminator work together, the model achieves higher accuracy and significantly faster convergence than traditional distillation or two-player GAN approaches.

What Problem Does This Paper Address?

In the world of machine learning, we often face a "resource gap." During training, we might have access to massive computational power and "privileged information"—extra features like text descriptions for images that won't be available when the model is actually deployed on a smartphone or edge device.

To bridge this gap, researchers typically use Knowledge Distillation (KD), where a large "teacher" model trains a small "student" classifier. However, if the teacher isn't perfect, the student's performance is capped by the teacher's biases. An alternative is using Generative Adversarial Networks (GANs), where a classifier tries to fool a discriminator. While GANs can theoretically learn the "true" data distribution, they are notoriously difficult to train, often suffering from high-variance gradients and agonizingly slow convergence when dealing with discrete labels.

Methods: The KDGAN Framework

The authors introduce KDGAN, a three-player minimax game designed to leverage the strengths of both KD and GANs while mitigating their weaknesses.

The KDGAN framework featuring a Classifier, Teacher, and Discriminator.

The system consists of three distinct neural networks:

  1. The Classifier ($C$): The lightweight model we want to deploy.
  2. The Teacher ($T$): A high-capacity model with access to privileged information.
  3. The Discriminator ($D$): A judge that tries to distinguish real labels from those generated by the Classifier or Teacher.

The "Secret Sauce"

  • Mutual Learning: Unlike standard KD where the teacher is static, in KDGAN, the classifier and teacher learn from each other via distillation losses. This "student-teacher interaction" helps both models reach an agreement on the data distribution.
  • Adversarial Synergy: Both the classifier and teacher act as generators trying to fool the discriminator. This ensures the classifier doesn't just mimic the teacher's mistakes but actually aims for the true data distribution.
  • The Gumbel-Max Trick: To solve the problem of "non-differentiable" discrete labels (which usually breaks backpropagation in GANs), the authors use a Concrete distribution relaxation. This allows them to generate continuous samples, providing low-variance gradients that speed up training.

Datasets and Tasks

The authors evaluated KDGAN on two primary tasks:

  • Deep Model Compression: Using the classic MNIST and CIFAR-10 datasets. They tested scenarios with limited training data (as few as 100 images) to see how well the model generalizes.
  • Image Tag Recommendation: Using the YFCC100M (Yahoo Flickr Creative Commons) dataset. Here, the "privileged information" for the teacher consisted of image titles and user comments, while the classifier only had access to the images themselves.

Key Insights and Ideas

  • Complementary Strengths: The authors realized that KD and GANs are perfect partners. KD provides a stable, low-variance training signal (fast but potentially biased), while the GAN component provides an unbiased signal (accurate but noisy). Together, they balance each other out.
  • Teacher-Student Interaction: Interestingly, the paper shows that the teacher actually benefits from learning from the student. This mutual distillation helps the entire system converge to a better equilibrium.
  • Variance Control: The use of the Gumbel-Max trick wasn't just a technical necessity; it was a performance booster. By smoothing the discrete labels into a continuous space, they reduced gradient variance, making the adversarial training much more stable.

Key Results

KDGAN demonstrated significant improvements over existing baselines in both accuracy and training efficiency.

Accuracy vs. Training Epochs demonstrating KDGAN's rapid convergence.
  • Superior Accuracy: On MNIST with only 100 training samples, KDGAN achieved a 5.31% performance gain over the best KD-based baseline.
  • Faster Convergence: In the MNIST experiments, KDGAN converged in roughly 25 epochs, whereas a naive GAN approach (NaGAN) required about 135 epochs. This represents a massive reduction in training time and highlights the stabilizing effect of the distillation loss.
  • Better Tagging: In the image tagging task, KDGAN outperformed state-of-the-art methods like REXMP across all metrics (Precision, F-score, and MAP), proving that the teacher's "soft labels" capture rich semantic relationships between tags that hard labels miss.

Limitations and Future Work

While KDGAN is robust, it does introduce several hyperparameters ($\alpha, \beta, \gamma$) that control the balance between distillation and adversarial losses. The authors noted that while the model is relatively stable, a high value for $\gamma$ (forcing the teacher to mimic a noisy student too closely) can degrade performance.

Future Work: The authors suggest exploring adaptive methods to automatically determine these hyperparameters during training, potentially removing the need for manual tuning and making the framework even more "plug-and-play" for different architectures.

Summary updated

Abstract

Knowledge distillation (KD) aims to train a lightweight classifier suitable to provide accurate inference with constrained resources in multi-label learning. Instead of directly consuming feature-label pairs, the classifier is trained by a teacher, i.e., a high-capacity model whose training may be resource-hungry. The accuracy of the classifier trained this way is usually suboptimal because it is difficult to learn the true data distribution from the teacher. An alternative method is to adversarially train the classifier against a discriminator in a two-player game akin to generative adversarial networks (GAN), which can ensure the classifier to learn the true data distribution at the equilibrium of this game. However, it may take excessively long time for such a two-player game to reach equilibrium due to high-variance gradient updates. To address these limitations, we propose a three-player game named KDGAN consisting of a classifier, a teacher, and a discriminator. The classifier and the teacher learn from each other via distillation losses and are adversarially trained against the discriminator via adversarial losses. By simultaneously optimizing the distillation and adversarial losses, the classifier will learn the true data distribution at the equilibrium. We approximate the discrete distribution learned by the classifier (or the teacher) with a concrete distribution. From the concrete distribution, we generate continuous samples to obtain low-variance gradient updates, which speed up the training. Extensive experiments using real datasets confirm the superiority of KDGAN in both accuracy and training speed.
BibTeX
@inproceedings{wang2018kdgan,
  author = {Xiaojie Wang and Rui Zhang and Yu Sun and Jianzhong Qi},
  title = {{KDGAN: Knowledge Distillation with Generative Adversarial Networks}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2018},
}
Figure