Improving GANs with A Dynamic Discriminator

Ceyuan Yang, Yujun Shen, Yinghao Xu, Deli Zhao, Bo Dai, Bolei Zhou

Abstract

The discriminator plays a vital role in training generative adversarial networks (GANs) by distinguishing real and synthesized samples. While the real data distribution remains constant, the synthesis distribution varies as the generator evolves, fundamentally changing the classification task for the discriminator. We propose DynamicD, a training strategy that adjusts the discriminator's capacity on-the-fly to accommodate this time-varying task. Our empirical study confirms that DynamicD improves synthesis performance without additional computational costs. We develop two capacity-adjusting schemes: progressively increasing capacity for sufficient data regimes and gradually decreasing layer width for limited data regimes to mitigate overfitting. DynamicD is synergistic with existing discriminator-improving approaches and demonstrates substantial gains across 2D and 3D-aware image synthesis tasks.

The Challenge: A Moving Target

In standard GAN frameworks, the discriminator maintains a fixed architecture. However, the difficulty of its task—telling real images from fake ones—is non-stationary. Early in training, the generator produces crude noise, making the discriminator's job trivial. As the generator matures, the "fake" samples become increasingly sophisticated, requiring a more powerful discriminator to provide useful gradients.

This static approach leads to two primary failure modes:

  1. Under-fitting (Sufficient Data): When training data is plentiful, a fixed-capacity discriminator may eventually lack the representational power to keep pace with a highly evolved generator, stalling progress.
  2. Over-fitting (Limited Data): In low-data regimes, a large discriminator often "memorizes" the small training set rather than learning generalizable features. This leads to a collapse in synthesis quality as the discriminator becomes too proficient at identifying specific training samples.
Figure 1: Training progress on sufficient vs. limited data. (a) With sufficient data, FID decreases as samples become harder to distinguish. (b) With limited data, the discriminator eventually overfits, causing FID to rise as samples become easily distinguishable.

Methodology: DynamicD

To address these issues, DynamicD adjusts the discriminator's width (number of filters) during training. This alignment between model complexity and task difficulty ensures stable competition throughout the GAN process.

1. Increasing Capacity (Sufficient Data)

For large datasets, the discriminator starts with a reduced width (e.g., 50% of the baseline). As training progresses, new filters are initialized and added linearly until the model reaches full capacity. This "growth" phase mirrors the increasing complexity of the generator's output.

2. Decreasing Capacity (Limited Data)

To combat memorization in data-scarce environments, the discriminator starts at full capacity and gradually shrinks its width. This is achieved by randomly dropping a subset of filters at each step, acting as a form of weight-level dropout. This prevents the model from latching onto specific training samples and encourages the learning of more robust features.

Crucially, DynamicD incurs zero additional computational cost because the model is always smaller than or equal to the baseline size during training.

Experimental Results

The researchers evaluated DynamicD across several benchmarks, including FFHQ (faces), LSUN (scenes), and Carla (3D-aware synthesis).

  • Low-Data Regimes: On the FFHQ 0.1k dataset, DynamicD achieved a massive improvement, reducing the Fréchet Inception Distance (FID) from 179.21 (StyleGAN2 baseline) to 50.37.
  • High-Data Regimes: Even on massive datasets like LSUN Bedroom, where baselines are already strong, DynamicD pushed the FID from 5.65 down to 4.01.
  • 3D-Aware Synthesis: Applied to StyleNeRF on the Carla dataset, it improved FID from 53.87 to 47.42, demonstrating that the strategy generalizes beyond 2D pixels.

Conclusion

DynamicD offers a simple yet powerful paradigm shift: matching the discriminator's capacity to the evolving difficulty of the GAN training task. By treating the discriminator as a dynamic component rather than a static one, we can significantly improve image quality and training stability across diverse data regimes and architectures. Future work may explore automated scheduling via AutoML or extending these principles to depth-based or Transformer-based architectures.

Summary updated

Abstract

Discriminator plays a vital role in training generative adversarial networks (GANs) via distinguishing real and synthesized samples. While the real data distribution remains the same, the synthesis distribution keeps varying because of the evolving generator, and thus effects a corresponding change to the bi-classification task for the discriminator. We argue that a discriminator with an on-the-fly adjustment on its capacity can better accommodate such a time-varying task. A comprehensive empirical study confirms that the proposed training strategy, termed as DynamicD, improves the synthesis performance without incurring any additional computation cost or training objectives. Two capacity adjusting schemes are developed for training GANs under different data regimes: i) given a sufficient amount of training data, the discriminator benefits from a progressively increased learning capacity, and ii) when the training data is limited, gradually decreasing the layer width mitigates the over-fitting issue of the discriminator. Experiments on both 2D and 3D-aware image synthesis tasks conducted on a range of datasets substantiate the generalizability of our DynamicD as well as its substantial improvement over the baselines. Furthermore, DynamicD is synergistic to other discriminator-improving approaches (including data augmentation, regularizers, and pre-training), and brings continuous performance gain when combined for learning GANs.
BibTeX
@inproceedings{yang2022improving,
  author = {Ceyuan Yang and Yujun Shen and Yinghao Xu and Deli Zhao and Bo Dai and Bolei Zhou},
  title = {{Improving GANs with A Dynamic Discriminator}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=MwqF0IkawQ-},
}
Figure