Momentum Adversarial Distillation: Handling Large Distribution Shifts in Data-Free Knowledge Distillation

Kien Do, Hung Le, Dung Nguyen, Dang Nguyen, Haripriya Harikumar, Truyen Tran, Santu Rana, Svetha Venkatesh

TL;DR

This paper introduces Momentum Adversarial Distillation (MAD), a novel technique for Data-Free Knowledge Distillation (DFKD) that tackles the challenge of distribution shifts in synthetic data. By employing an Exponential Moving Average (EMA) of the generator, MAD helps the student network retain knowledge and avoid forgetting, leading to improved performance, especially on large datasets.

What Problem Does This Paper Address?

The core problem addressed is catastrophic forgetting in Data-Free Knowledge Distillation (DFKD). In DFKD, a generator synthesizes data to train a student network from a teacher network, as the original training data is unavailable. When the generator and student are trained adversarially, the distribution of the synthetic data can shift dramatically between training steps. This rapid shift can cause the student network to forget previously learned knowledge, hindering its performance. Existing adversarial DFKD methods struggle with this "distribution shift" problem, leading to performance degradation.

Methods

The paper proposes Momentum Adversarial Distillation (MAD). The key innovation is the introduction of an Exponential Moving Average (EMA) generator, denoted as $\tilde{G}$. This EMA generator acts as a stable, historical snapshot of the generator $G$.

Here's how it works:

  • Standard Adversarial DFKD: Typically involves a generator $G$ and a student $S$ trained adversarially. $G$ generates synthetic data, and $S$ tries to mimic the teacher $T$'s predictions on this data. The loss function encourages $G$ to generate samples that are hard for $S$ to classify correctly according to $T$.
  • MAD's Enhancement: MAD augments the training of $S$ by using synthetic samples from both the current generator $G$ and the EMA generator $\tilde{G}$. The loss function for $S$ is a weighted sum of distillation losses on samples from $G$ and $\tilde{G}$.
  • EMA Generator Update: The EMA generator $\tilde{G}$ is updated using an exponential moving average formula: $\theta_{\tilde{G}} = \alpha \cdot \theta_{\tilde{G}_{old}} + (1 - \alpha) \cdot \theta_G$, where $\alpha$ is the momentum coefficient. A high $\alpha$ means $\tilde{G}$ changes slowly, preserving past knowledge.
  • Class-Conditional Generator for Large Datasets: To improve performance on large datasets like ImageNet, MAD also proposes a class-conditional generator. This generator takes a noise vector and a class embedding as input, helping to suppress spurious solutions and ensure more stable training. A specific loss term, $L_{NLL}$, is introduced to encourage the generator to focus on predicting the correct class.
  • BatchNorm Moment Matching (BNmm): For teachers with BatchNorm layers, an additional loss term is used to match the running statistics (mean and variance) of the synthetic data with those of the teacher's BatchNorm layers, further improving data synthesis quality.

Datasets

The paper evaluates MAD on a variety of image classification datasets:

  • Small Datasets: CIFAR10, CIFAR100, TinyImageNet
  • Large Datasets: ImageNet, Places365, Food101

Key Insights and Ideas

  • EMA as a Stabilizer: The core insight is that an EMA of the generator acts as a "memory" of past distributions. By training on samples from both the current and EMA generators, the student is less likely to forget previous knowledge when the current generator's distribution shifts rapidly.
  • Preventing "Forgetting" in DFKD: The paper frames the problem as a form of catastrophic forgetting, similar to continual learning, but specifically within the DFKD context. MAD directly addresses this by stabilizing the learning process.
  • Class-Conditional Generators for Scalability: The introduction of class-conditional generators with a specific regularization loss ($L_{NLL}$) is crucial for enabling MAD to perform well on large-scale datasets where unconditional generators might collapse due to a vast number of spurious solutions.
  • The "Sum" Generator's Advantage: The "sum" approach for conditional generators (adding noise and class embeddings) is found to be more stable and effective than concatenation, likely due to better absorption of stochasticity.

Key Results

  • Superior Performance: MAD consistently outperforms related baselines like ABM and DFKD-Mem across all tested datasets, especially on larger ones.
  • Significant Accuracy Gains: On ImageNet, MAD achieves approximately 4.2% and 2.2% higher accuracy than ABM and DFKD-Mem, respectively.
  • State-of-the-Art Potential: In some cases, MAD achieves state-of-the-art results.
  • Efficiency: Compared to methods like DeepInversion, MAD requires significantly fewer training iterations for comparable performance on ImageNet.
  • Generator Stability: The proposed class-conditional generator (especially the "sum" variant) demonstrates much more stable training and better performance on large datasets compared to unconditional generators.

Limitations and Future Work

  • Increased Training Time: The current implementation of MAD requires an additional forward pass through the EMA generator for each student training step, increasing training time by about 40% compared to ABM.
  • Future Work: The authors suggest addressing the increased training time by storing synthetic samples from both generators in a buffer before student training. They also note that the EMA generator concept is generalizable to other areas like continual learning and domain adaptation.

Summary updated

Abstract

Data-free Knowledge Distillation (DFKD) has attracted attention recently thanks to its appealing capability of transferring knowledge from a teacher network to a student network without using training data. The main idea is to use a generator to synthesize data for training the student. As the generator gets updated, the distribution of synthetic data will change. Such distribution shift could be large if the generator and the student are trained adversarially, causing the student to forget the knowledge it acquired at previous steps. To alleviate this problem, we propose a simple yet effective method called Momentum Adversarial Distillation (MAD) which maintains an exponential moving average (EMA) copy of the generator and uses synthetic samples from both the generator and the EMA generator to train the student. Since the EMA generator can be considered as an ensemble of the generator's old versions and often undergoes a smaller change in updates compared to the generator, training on its synthetic samples can help the student recall the past knowledge and prevent the student from adapting too quickly to new updates of the generator. Our experiments on six benchmark datasets including big datasets like ImageNet and Places365 demonstrate the superior performance of MAD over competing methods for handling the large distribution shift problem. Our method also compares favorably to existing DFKD methods and even achieves state-of-the-art results in some cases.
BibTeX
@inproceedings{do2022momentum,
  author = {Kien Do and Hung Le and Dung Nguyen and Dang Nguyen and Haripriya Harikumar and Truyen Tran and Santu Rana and Svetha Venkatesh},
  title = {{Momentum Adversarial Distillation: Handling Large Distribution Shifts in Data-Free Knowledge Distillation}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=MZbO43Xf4C},
}
Figure