Self-Supervised Generative Adversarial Compression

Chong Yu, Jeff PoolNeurIPS 2020 · OpenReview

TL;DR

Standard model compression techniques like pruning and distillation often fail when applied to complex Generative Adversarial Networks (GANs), leading to distorted images or "mode collapse." This paper introduces a self-supervised compression framework that repurposes the original, pre-trained discriminator to guide the training of a compressed generator, achieving high levels of sparsity (up to 90%) with minimal loss in image quality.

The Challenge of Compressing GANs

Deep learning models are growing exponentially, but deploying them on edge devices or in real-time applications requires them to be smaller and faster. While the industry has become highly proficient at compressing "discriminative" models (like those used for image classification), GANs present a unique challenge.

GANs rely on a delicate Nash Equilibrium between a generator (which creates images) and a discriminator (which catches fakes). The authors point out that standard pruning—simply cutting out "unimportant" weights—breaks this balance. In complex tasks like style transfer or face attribute editing, traditional compression leads to "facial artifacts" (distorted faces) or "mode collapse" (where the model starts outputting the same blurry mess regardless of the input). Surprisingly, the authors found that even when the training loss curves look healthy, the actual visual output can be entirely unusable.

A Self-Supervised Compression Framework

To solve this, the authors propose a Self-Supervised Generator Compression framework. The core innovation lies in how they handle the loss function during the fine-tuning of a pruned model.

Architecture of the Self-Supervised Generator Compression framework showing the Dual Consistency Loss.
  1. The "Self-Supervisor": Instead of discarding the discriminator after the original training is done, they keep it. This pre-trained discriminator ($D_O$) acts as a "quantitative subjective judge." If the discriminator cannot tell the difference between the original generator's output and the compressed generator's output, then the compression is deemed successful.
  2. Dual Consistency Loss: They introduce two main loss terms to guide the process:
  3. Generative Consistent Loss ($L_{GC}$): This ensures the compressed generator ($G_C$) produces outputs similar to the original generator ($G_O$) by comparing their internal loss terms (generation, classification, and reconstruction).
  4. Discriminative Consistent Loss ($L_{DC}$): This measures how the original discriminator perceives the outputs of both the original and compressed generators.
  5. Fine-Grained Sparsity: The paper explores different "granularities" of pruning. They find that while "filter pruning" (removing entire chunks of the network) is easier for hardware to run, it destroys GAN performance. Instead, they advocate for fine-grained structured sparsity, specifically the 2:4 pattern (where 2 out of every 4 weights are zero). This provides a middle ground that maintains high quality while allowing for hardware acceleration on modern GPUs like the NVIDIA A100.

Key Insights

  • Loss Curves Lie: One of the most striking findings is that for GAN compression, traditional loss curves are "false positives." A model can show perfect convergence while producing "color artifacts" or "no faces." Subjective visual quality and FID scores are the only reliable metrics.
  • The Discriminator is a Teacher: In standard distillation, you usually use a "teacher" model to train a "student." Here, the discriminator—which was originally the generator's adversary—becomes its most effective tutor during the compression phase.
  • Entropy Matters: The authors argue that GANs are harder to compress than classifiers because they must maintain high entropy. A classifier reduces a high-resolution image to a single label (low entropy), but a GAN transforms an image into another high-resolution image (high entropy), leaving much less room for error in the compressed "information space."

Results and Performance

The researchers tested their method across a wide variety of generative tasks to prove its generality, including basic image synthesis (MNIST), multi-domain face attribute translation (CelebA), domain translation (Cityscapes), unpaired style transfer (Monet/Zebra), and image super-resolution (DIV2K).

Comparison of the proposed method (b) against 13 baselines, demonstrating superior qualitative results and FID scores.
  • Superior Quality: The proposed method outperformed 13 other baseline approaches (including AGP and standard distillation). While other methods resulted in mode collapse or severe artifacts at 50% sparsity, this method maintained "Good" qualitative results and the lowest FID scores.
  • High Sparsity: The framework remains robust even at 90% element-wise sparsity. While some fine details fade at extreme levels, the core structure and style of the images remain intact.
  • Efficiency: The self-supervised compression process is remarkably fast, requiring only 1-10% of the original training time to recover the performance of a pruned model.
  • Hardware Ready: By using the 2:4 structured sparsity pattern, the authors achieved results indistinguishable from unstructured pruning, providing a clear path to 2x speedups on compatible hardware.

Limitations and Future Work

The authors acknowledge they haven't yet pushed the limits of "extremely aggressive" compression (e.g., combining pruning with heavy quantization or weight sharing). They also note that while the 0.5 hyperparameter for $\lambda$ worked across all their tests, further tuning might be needed for even more exotic architectures. Future work will likely look at extending this self-supervised approach to newer, even larger generative models like Transformers-based generators.

Summary updated

Abstract

Deep learning’s success has led to larger and larger models to handle more and more complex tasks; trained models often contain millions of parameters. These large models are compute- and memory-intensive, which makes it a challenge to deploy them with latency, throughput, and storage constraints. Some model compression methods have been successfully applied to image classification and detection or language models, but there has been very little work compressing generative adversarial networks (GANs) performing complex tasks. In this paper, we show that a standard model compression technique, weight pruning and knowledge distillation, cannot be applied to GANs using existing methods. We then develop a self-supervised compression technique which uses the trained discriminator to supervise the training of a compressed generator. We show that this framework has compelling performance to high degrees of sparsity, can be easily applied to new tasks and models, and enables meaningful comparisons between different compression granularities.
BibTeX
@inproceedings{yu2020self,
  author = {Chong Yu and Jeff Pool},
  title = {{Self-Supervised Generative Adversarial Compression}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=AQUYfjnjeQi},
}
Figure