TL;DR
Generative Adversarial Networks (GANs) typically use a discriminator with a fixed capacity, even though the difficulty of its task changes as the generator improves. This paper introduces "DynamicD," a training strategy that adjusts the discriminator's capacity on-the-fly—increasing it for large datasets to prevent under-fitting, and decreasing it for small datasets to prevent over-fitting. This simple, compute-free tweak significantly improves image synthesis quality and is highly compatible with existing GAN enhancements.
What Problem Does This Paper Address?
In the two-player game of a GAN, the generator tries to create realistic fake images, while the discriminator tries to classify images as real or fake. In standard image classification, the dataset is static. But in GAN training, the "fake" data distribution is constantly shifting as the generator gets better at its job.
Because the generator evolves, the discriminator's bi-classification task is time-varying. Early in training, it's easy to tell noise from a real photo. Later on, it becomes incredibly difficult. Despite this dynamic environment, almost all GANs use a discriminator with a fixed capacity (a static neural network architecture).
The authors argue that a fixed capacity is suboptimal. If the capacity is too high for a small dataset, the discriminator simply memorizes the real images (over-fitting). If the capacity is too low for a massive dataset, it fails to keep up with a highly capable generator (under-fitting). The paper asks: Can dynamically adjusting the discriminator's capacity during training solve these issues?
Methods
The authors propose DynamicD, a method to adjust the discriminator's layer width on-the-fly during training. They introduce two distinct schemes based on the size of the training data:
- Increasing Capacity (for sufficient data): When data is plentiful, the discriminator needs more power later in training to distinguish highly realistic fakes from real images. The model starts at a reduced capacity (e.g., half the standard layer width). As training progresses, newly initialized filters are progressively added to the layers until the network reaches full size.
- Decreasing Capacity (for limited data): When data is scarce, the discriminator is prone to memorizing the dataset late in training. To combat this, the model starts at full capacity. Over time, the network width is progressively shrunk by randomly dropping a subset of filters. This acts as a "weight-level" dropout shared across the batch, heavily regularizing the network against over-fitting.
Crucially, because the network is smaller than the baseline for portions of the training process, DynamicD actually reduces overall computational overhead while requiring no additional loss functions.
Datasets
The authors evaluated DynamicD across a variety of 2D and 3D image synthesis benchmarks:
- FFHQ (256x256): Human faces. Used extensively to test both limited (100 to 2,000 images) and sufficient (up to 140K augmented images) data regimes.
- AFHQ-v2 (512x512): Cats, dogs, and wildlife (treated as limited data).
- LSUN (256x256): Church, bridge, and bedroom scenes (treated as sufficient data, up to 3 million images).
- Carla: A synthetic dataset of rendered cars used to evaluate 3D-aware GANs.
- MetFaces: Used to test the compatibility of DynamicD with transfer learning/fine-tuning.
Key Insights and Ideas
- Task difficulty is dynamic, so the model should be too: The core "aha" moment is treating GAN training as a moving target. By aligning the discriminator's capacity with the current skill level of the generator, the adversarial game remains balanced.
- Data scale dictates the strategy: You cannot use a one-size-fits-all dynamic approach. Increasing capacity on a small dataset actually exacerbates over-fitting, while decreasing capacity on a large dataset starves the model of needed complexity.
- Standard discriminators have redundant parameters: Even when starting at half capacity and growing to full size, DynamicD outperforms a static full-capacity model on large datasets. This implies that having all parameters active from step one might actually hinder optimal training.
- Synergy with existing methods: Dynamic capacity isn't mutually exclusive with popular data augmentation techniques (like ADA) or regularizers. Because it operates on the architecture itself, it stacks cleanly with other methods for cumulative gains.
Key Results
- Extreme Low-Data SOTA: On FFHQ with just 100 training images, DynamicD slashed the Fréchet Inception Distance (FID) from 179.21 (baseline StyleGAN2) to 50.37. It outperformed heavy data augmentation baselines like DiffAugment (61.91) and ADA (82.17) in this regime.
- Sufficient Data Improvements: On the massive 3-million image LSUN Bedroom dataset, DynamicD improved FID from 5.65 to 4.01.
- 3D-Aware GANs: When applied to StyleNeRF on a limited 2K subset of FFHQ, DynamicD improved FID massively from 73.50 to 23.29.
- Compatibility: When combined with ADA on a 2K FFHQ dataset, FID dropped from 15.62 (ADA alone) to 14.56 (ADA + DynamicD).
Limitations and Future Work
The authors note a few limitations to their approach:
- Architectural constraints: DynamicD was only tested on CNN-based discriminators. It is unclear how well this translates to Transformer-based discriminators (like TransGAN).
- Adjustment dimensions: The current method only adjusts network width (number of filters). Adjusting network depth (number of layers) remains unexplored.
- Manual heuristics: The linear scaling of capacity (up or down) is a manual heuristic. The authors suggest that exploring self-adjusting or AutoML strategies could yield even more optimal capacity tuning.
- Lack of theory: While empirically strong, the paper lacks a rigorous theoretical study explaining exactly how these dynamic adjustments interact with the minimax game's convergence.
Summary updated