DGL-GAN: Discriminator Guided Learning for GAN Compression

Yuesong Tian, Li Shen, Xiang Tian, Dacheng Tao, Zhifeng Li, Wei Liu, Yaowu Chen

TL;DR

DGL-GAN is a novel framework that compresses large-scale Generative Adversarial Networks (GANs) like StyleGAN2 and BigGAN by using a pretrained "teacher" discriminator to guide a smaller "student" generator. By ditching traditional feature-matching distillation in favor of pure adversarial guidance, the authors achieve state-of-the-art compression results and even push uncompressed models to new performance peaks, such as an FID of 2.65 on FFHQ.

What Problem Does This Paper Address?

Modern GANs produce stunning, high-resolution images, but they are computationally gluttonous. Models like BigGAN and StyleGAN2 have tens of millions of parameters, making them difficult to deploy on mobile or edge devices.

Compressing these "vanilla" (noise-to-image) GANs is notoriously harder than compressing standard image classifiers for several reasons:

  1. Parameter Sensitivity: Generators are fragile; pruning even a small fraction of weights can catastrophically degrade image quality (see Figure 2).
  2. Lack of Ground Truth: In noise-to-image tasks, there is no "correct" target image for a given random input. Two perfectly valid generators might produce completely different images from the same latent code (see Figure 3), making traditional pixel-wise or feature-wise distillation ineffective.
  3. Training Instability: The minimax nature of GAN training is already unstable; adding a strong teacher model can easily overwhelm a small student, leading to training divergence.
Figure 3 illustrates that different generators produce distinct images from the same latent code, meaning there is no single "ground truth" image to align with.

Methods

The core idea of Discriminator Guided Learning (DGL-GAN) is to ignore the teacher generator entirely and instead harvest the knowledge embedded in a pretrained teacher discriminator ($\bar{D}$).

The DGL Objective

The student generator ($G$) is trained to fool two entities simultaneously: its own student discriminator ($D$) and the fixed, high-performing teacher discriminator ($\bar{D}$). The objective function adds a regularization term based on the teacher's feedback: $$\min_G \max_D V(G, D) = Adv(G, D) + \lambda * Adv(G, \bar{D})$$ where $\lambda$ balances the two signals.

Two-Stage Training

To solve the instability problem, the authors propose a two-stage strategy:

  • Stage I: Train the narrow student GAN from scratch using standard adversarial loss. This provides a stable initialization.
  • Stage II: Fine-tune the student generator using the additional supervision from the teacher discriminator.

Width Reduction

Instead of complex pruning or architecture search, DGL-GAN simply reduces the "width" (number of channels) of the network using a channel multiplier. This approach is highly efficient for GPU inference compared to sparse methods.

Datasets

The authors evaluated DGL-GAN on three standard high-resolution benchmarks:

  • FFHQ (Flickr-Faces-HQ): 70,000 high-quality face images at 1024x1024 resolution.
  • LSUN-church: 126,227 images of churches at 256x256 resolution.
  • ImageNet: Over 1.2 million diverse images across 1,000 categories at 128x128 resolution.

Key Insights and Ideas

  • The Discriminator as a Distribution Map: The authors argue that a well-trained teacher discriminator implicitly "knows" the target data distribution better than the generator does. Its gradients provide a clearer, more meaningful supervision signal for the student (see Figure 5).
  • Alignment is a Trap: Previous methods tried to force student generators to mimic teacher generators. DGL-GAN shows that because there is no unique ground truth for a latent code, this "imitation" actually hampers performance.
  • The "Strong Teacher" Problem: On complex datasets like ImageNet, a pretrained teacher discriminator is too good. If introduced too early, it immediately identifies all student images as fake, providing no useful gradient for learning. The two-stage approach is essential to bridge this gap.
Figure 5 visualizes how the teacher discriminator provides clearer, more indicative gradients (bottom row) compared to the student discriminator (top row).

Key Results

DGL-GAN achieves state-of-the-art (SOTA) trade-offs between computation (MACs) and image quality (FID), as shown in Figure 1.

  • StyleGAN2 Compression: On FFHQ, DGL-GAN with a 1/2 channel multiplier (roughly 1/3 of the parameters) achieved an FID of 2.97, nearly matching the original model's 2.84 while being significantly faster.
  • BigGAN Compression: On ImageNet, it achieved a comparable Inception Score (IS) using only 1/4 of the parameters.
  • Boosting Uncompressed Models: Perhaps most surprisingly, applying DGL-GAN to a full-sized StyleGAN2 model improved its FID from 2.84 to 2.65, setting a new SOTA that even surpasses the more recent StyleGAN3.
  • Inference Speed: The compressed models achieved up to a 4.38x acceleration on a Tesla V100 GPU.
Figure 1 shows the MACs-Performance curves, demonstrating that DGL-GAN consistently outperforms existing compression methods like Anycost-GAN and CA-GAN.

Limitations and Future Work

  • Manual Scheduling: The transition point between the two training stages (the number of iterations $T$ and $S$) is currently set manually. The authors suggest that an automatic mechanism to determine these thresholds would be a valuable improvement.
  • Discriminator Precision: The method relies on the teacher discriminator being close to the "global optimal." Future work could explore ways to obtain even more precise discriminators to further boost student performance.
  • Theoretical Grounding: While empirically successful, a deeper theoretical analysis of why DGL-GAN boosts uncompressed models could provide further insights into GAN optimization.

Summary updated

Abstract

Generative Adversarial Networks (GANs) with high computation costs, e.g., BigGAN and StyleGAN2, have achieved remarkable results in synthesizing high-resolution images from random noise. Reducing the computation cost of GANs while keeping generating photo-realistic images is a challenging field. In this work, we propose a novel yet simple {\bf D}iscriminator {\bf G}uided {\bf L}earning approach for compressing vanilla {\bf GAN}, dubbed {\bf DGL-GAN}. Motivated by the phenomenon that the teacher discriminator may contain some meaningful information about both real images and fake images, we merely transfer the knowledge from the teacher discriminator via the adversarial interaction between the teacher discriminator and the student generator. We apply DGL-GAN to compress the two most representative large-scale vanilla GANs, i.e., StyleGAN2 and BigGAN. Experiments show that DGL-GAN achieves state-of-the-art (SOTA) results on both StyleGAN2 and BigGAN. Moreover, DGL-GAN is also effective in boosting the performance of original uncompressed GANs. Original uncompressed StyleGAN2 boosted with DGL-GAN achieves FID 2.65 on FFHQ, which achieves a new state-of-the-art performance. Code and models are available at \url{https://github.com/yuesongtian/DGL-GAN}
BibTeX
@article{tian2025dgl,
  author = {Yuesong Tian and Li Shen and Xiang Tian and Dacheng Tao and Zhifeng Li and Wei Liu and Yaowu Chen},
  title = {{DGL-GAN: Discriminator Guided Learning for GAN Compression}},
  journal = {The Visual Computer},
  year = {2025},
  doi = {10.1007/s00371-024-03682-y},
}
Figure