TL;DR
DGL-GAN is a novel framework that compresses large-scale Generative Adversarial Networks (GANs) like StyleGAN2 and BigGAN by using a pretrained "teacher" discriminator to guide a smaller "student" generator. By ditching traditional feature-matching distillation in favor of pure adversarial guidance, the authors achieve state-of-the-art compression results and even push uncompressed models to new performance peaks, such as an FID of 2.65 on FFHQ.
What Problem Does This Paper Address?
Modern GANs produce stunning, high-resolution images, but they are computationally gluttonous. Models like BigGAN and StyleGAN2 have tens of millions of parameters, making them difficult to deploy on mobile or edge devices.
Compressing these "vanilla" (noise-to-image) GANs is notoriously harder than compressing standard image classifiers for several reasons:
- Parameter Sensitivity: Generators are fragile; pruning even a small fraction of weights can catastrophically degrade image quality (see Figure 2).
- Lack of Ground Truth: In noise-to-image tasks, there is no "correct" target image for a given random input. Two perfectly valid generators might produce completely different images from the same latent code (see Figure 3), making traditional pixel-wise or feature-wise distillation ineffective.
- Training Instability: The minimax nature of GAN training is already unstable; adding a strong teacher model can easily overwhelm a small student, leading to training divergence.
Methods
The core idea of Discriminator Guided Learning (DGL-GAN) is to ignore the teacher generator entirely and instead harvest the knowledge embedded in a pretrained teacher discriminator ($\bar{D}$).
The DGL Objective
The student generator ($G$) is trained to fool two entities simultaneously: its own student discriminator ($D$) and the fixed, high-performing teacher discriminator ($\bar{D}$). The objective function adds a regularization term based on the teacher's feedback: $$\min_G \max_D V(G, D) = Adv(G, D) + \lambda * Adv(G, \bar{D})$$ where $\lambda$ balances the two signals.
Two-Stage Training
To solve the instability problem, the authors propose a two-stage strategy:
- Stage I: Train the narrow student GAN from scratch using standard adversarial loss. This provides a stable initialization.
- Stage II: Fine-tune the student generator using the additional supervision from the teacher discriminator.
Width Reduction
Instead of complex pruning or architecture search, DGL-GAN simply reduces the "width" (number of channels) of the network using a channel multiplier. This approach is highly efficient for GPU inference compared to sparse methods.
Datasets
The authors evaluated DGL-GAN on three standard high-resolution benchmarks:
- FFHQ (Flickr-Faces-HQ): 70,000 high-quality face images at 1024x1024 resolution.
- LSUN-church: 126,227 images of churches at 256x256 resolution.
- ImageNet: Over 1.2 million diverse images across 1,000 categories at 128x128 resolution.
Key Insights and Ideas
- The Discriminator as a Distribution Map: The authors argue that a well-trained teacher discriminator implicitly "knows" the target data distribution better than the generator does. Its gradients provide a clearer, more meaningful supervision signal for the student (see Figure 5).
- Alignment is a Trap: Previous methods tried to force student generators to mimic teacher generators. DGL-GAN shows that because there is no unique ground truth for a latent code, this "imitation" actually hampers performance.
- The "Strong Teacher" Problem: On complex datasets like ImageNet, a pretrained teacher discriminator is too good. If introduced too early, it immediately identifies all student images as fake, providing no useful gradient for learning. The two-stage approach is essential to bridge this gap.
Key Results
DGL-GAN achieves state-of-the-art (SOTA) trade-offs between computation (MACs) and image quality (FID), as shown in Figure 1.
- StyleGAN2 Compression: On FFHQ, DGL-GAN with a 1/2 channel multiplier (roughly 1/3 of the parameters) achieved an FID of 2.97, nearly matching the original model's 2.84 while being significantly faster.
- BigGAN Compression: On ImageNet, it achieved a comparable Inception Score (IS) using only 1/4 of the parameters.
- Boosting Uncompressed Models: Perhaps most surprisingly, applying DGL-GAN to a full-sized StyleGAN2 model improved its FID from 2.84 to 2.65, setting a new SOTA that even surpasses the more recent StyleGAN3.
- Inference Speed: The compressed models achieved up to a 4.38x acceleration on a Tesla V100 GPU.
Limitations and Future Work
- Manual Scheduling: The transition point between the two training stages (the number of iterations $T$ and $S$) is currently set manually. The authors suggest that an automatic mechanism to determine these thresholds would be a valuable improvement.
- Discriminator Precision: The method relies on the teacher discriminator being close to the "global optimal." Future work could explore ways to obtain even more precise discriminators to further boost student performance.
- Theoretical Grounding: While empirically successful, a deeper theoretical analysis of why DGL-GAN boosts uncompressed models could provide further insights into GAN optimization.
Summary updated