Improving GANs with A Dynamic Discriminator

Ceyuan Yang, Yujun Shen, Yinghao Xu, Deli Zhao, Bo Dai, Bolei Zhou

TL;DR

Generative Adversarial Networks (GANs) typically use a discriminator with a fixed capacity, even though the difficulty of its task changes as the generator improves. This paper introduces "DynamicD," a training strategy that adjusts the discriminator's capacity on-the-fly—increasing it for large datasets to prevent under-fitting, and decreasing it for small datasets to prevent over-fitting. This simple, compute-free tweak significantly improves image synthesis quality and is highly compatible with existing GAN enhancements.

What Problem Does This Paper Address?

In the two-player game of a GAN, the generator tries to create realistic fake images, while the discriminator tries to classify images as real or fake. In standard image classification, the dataset is static. But in GAN training, the "fake" data distribution is constantly shifting as the generator gets better at its job.

Training progress on sufficient vs. limited data

Because the generator evolves, the discriminator's bi-classification task is time-varying. Early in training, it's easy to tell noise from a real photo. Later on, it becomes incredibly difficult. Despite this dynamic environment, almost all GANs use a discriminator with a fixed capacity (a static neural network architecture).

The authors argue that a fixed capacity is suboptimal. If the capacity is too high for a small dataset, the discriminator simply memorizes the real images (over-fitting). If the capacity is too low for a massive dataset, it fails to keep up with a highly capable generator (under-fitting). The paper asks: Can dynamically adjusting the discriminator's capacity during training solve these issues?

Methods

The authors propose DynamicD, a method to adjust the discriminator's layer width on-the-fly during training. They introduce two distinct schemes based on the size of the training data:

  1. Increasing Capacity (for sufficient data): When data is plentiful, the discriminator needs more power later in training to distinguish highly realistic fakes from real images. The model starts at a reduced capacity (e.g., half the standard layer width). As training progresses, newly initialized filters are progressively added to the layers until the network reaches full size.
  2. Decreasing Capacity (for limited data): When data is scarce, the discriminator is prone to memorizing the dataset late in training. To combat this, the model starts at full capacity. Over time, the network width is progressively shrunk by randomly dropping a subset of filters. This acts as a "weight-level" dropout shared across the batch, heavily regularizing the network against over-fitting.

Crucially, because the network is smaller than the baseline for portions of the training process, DynamicD actually reduces overall computational overhead while requiring no additional loss functions.

Datasets

The authors evaluated DynamicD across a variety of 2D and 3D image synthesis benchmarks:

  • FFHQ (256x256): Human faces. Used extensively to test both limited (100 to 2,000 images) and sufficient (up to 140K augmented images) data regimes.
  • AFHQ-v2 (512x512): Cats, dogs, and wildlife (treated as limited data).
  • LSUN (256x256): Church, bridge, and bedroom scenes (treated as sufficient data, up to 3 million images).
  • Carla: A synthetic dataset of rendered cars used to evaluate 3D-aware GANs.
  • MetFaces: Used to test the compatibility of DynamicD with transfer learning/fine-tuning.

Key Insights and Ideas

  • Task difficulty is dynamic, so the model should be too: The core "aha" moment is treating GAN training as a moving target. By aligning the discriminator's capacity with the current skill level of the generator, the adversarial game remains balanced.
  • Data scale dictates the strategy: You cannot use a one-size-fits-all dynamic approach. Increasing capacity on a small dataset actually exacerbates over-fitting, while decreasing capacity on a large dataset starves the model of needed complexity.
  • Standard discriminators have redundant parameters: Even when starting at half capacity and growing to full size, DynamicD outperforms a static full-capacity model on large datasets. This implies that having all parameters active from step one might actually hinder optimal training.
  • Synergy with existing methods: Dynamic capacity isn't mutually exclusive with popular data augmentation techniques (like ADA) or regularizers. Because it operates on the architecture itself, it stacks cleanly with other methods for cumulative gains.

Key Results

  • Extreme Low-Data SOTA: On FFHQ with just 100 training images, DynamicD slashed the Fréchet Inception Distance (FID) from 179.21 (baseline StyleGAN2) to 50.37. It outperformed heavy data augmentation baselines like DiffAugment (61.91) and ADA (82.17) in this regime.
  • Sufficient Data Improvements: On the massive 3-million image LSUN Bedroom dataset, DynamicD improved FID from 5.65 to 4.01.
  • 3D-Aware GANs: When applied to StyleNeRF on a limited 2K subset of FFHQ, DynamicD improved FID massively from 73.50 to 23.29.
  • Compatibility: When combined with ADA on a 2K FFHQ dataset, FID dropped from 15.62 (ADA alone) to 14.56 (ADA + DynamicD).

Limitations and Future Work

The authors note a few limitations to their approach:

  • Architectural constraints: DynamicD was only tested on CNN-based discriminators. It is unclear how well this translates to Transformer-based discriminators (like TransGAN).
  • Adjustment dimensions: The current method only adjusts network width (number of filters). Adjusting network depth (number of layers) remains unexplored.
  • Manual heuristics: The linear scaling of capacity (up or down) is a manual heuristic. The authors suggest that exploring self-adjusting or AutoML strategies could yield even more optimal capacity tuning.
  • Lack of theory: While empirically strong, the paper lacks a rigorous theoretical study explaining exactly how these dynamic adjustments interact with the minimax game's convergence.

Summary updated

Abstract

Discriminator plays a vital role in training generative adversarial networks (GANs) via distinguishing real and synthesized samples. While the real data distribution remains the same, the synthesis distribution keeps varying because of the evolving generator, and thus effects a corresponding change of the bi-classification task assigned to the discriminator. We argue that a discriminator with an on-the-fly adjustment on its capacity can better accommodate such a time-varying task. A comprehensive empirical study confirms that the proposed training strategy, termed as DynamicD, improves the synthesis performance without incurring any additional computation cost or training objectives. Two capacity adjusting schemes are developed for training GANs under different data regimes: i) given a sufficient amount of training data, the discriminator benefits from a progressively increased learning capacity, and ii) when the training data is limited, gradually decreasing the layer width mitigates the over-fitting issue of the discriminator. Experiments on both 2D and 3D-aware image synthesis tasks conducted on a range of datasets substantiate the generalizability of our DynamicD as well as its substantial improvement over the baselines. Furthermore, DynamicD is synergistic to other discriminator-improving approaches (including data augmentation, regularizers, and pre-training), and brings continuous performance gain when combined with them for learning GANs. Code will be made publicly available.
BibTeX
@inproceedings{yang2022improving,
  author = {Ceyuan Yang and Yujun Shen and Yinghao Xu and Deli Zhao and Bo Dai and Bolei Zhou},
  title = {{Improving GANs with A Dynamic Discriminator}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=MwqF0IkawQ-},
}
Figure