TL;DR
Generative Adversarial Networks (GANs) often suffer from training instability, which standard Spectral Normalization attempts to fix using a rigid, one-size-fits-all constraint. This paper introduces ABCAS, a novel stabilizer that dynamically adjusts the discriminator's spectral norm bound based on the evolving distance between real and fake data distributions. This adaptive approach significantly improves training stability and image quality, especially on smaller datasets.
What Problem Does This Paper Address?
Training GANs is notoriously tricky. If the discriminator becomes too powerful too quickly, it stops providing useful gradients to the generator, leading to training divergence or mode collapse. Spectral Normalization (SN) is a widely accepted remedy that stabilizes training by bounding the Lipschitz constant of the discriminator, typically fixing the upper bound of its spectral norm to a constant $m = 1$.
However, the authors point out a major flaw in this approach: a static bound ignores the unique complexity and size of different datasets. Smaller datasets often cause the discriminator to overfit, requiring a tighter bound to keep it in check, whereas larger datasets can tolerate looser constraints. A fixed $m = 1$ constraint is rarely optimal and can still lead to training failure.
Methods
The authors propose ABCAS (Adaptive Bound Control of spectral norm as Automatic Stabilizer). Instead of fixing the spectral norm scaling factor $m$ to 1, ABCAS dynamically tunes it during training using the formula:
$$m = 0.9^r$$
where $r$ is an adaptive control variable.
The algorithm continuously monitors the distance between the discriminator's outputs for real and fake data:
$$dist = \max(C_{real}) - \min(C_{fake})$$
An exponential moving average of this distance ($dm$) is maintained. If this distance grows too large—indicating that the discriminator is easily separating real from fake data and potentially overfitting—the control variable $r$ increases. This drives $m$ down, tightening the spectral norm constraint and forcing the distribution gap to close. A hyperparameter $\beta$ acts as a threshold to scale this relationship, with $\beta = 4$ empirically found to yield the best results.
Datasets
The method was evaluated on three image datasets, all resized to $256 \times 256$ resolution:
- AFHQ-Cat: A relatively small dataset containing 5,165 high-quality cat faces.
- FFHQ: A large-scale dataset consisting of 70,000 human face images.
- FFHQ-10k: A smaller subset of FFHQ containing the first 10,000 images, specifically used to test the algorithm's performance under data-constrained conditions.
Key Insights and Ideas
- One Size Does Not Fit All: The optimal spectral norm bound is highly dataset-dependent. Standard Spectral Normalization ($m=1$) is rarely the ideal choice across different data scales.
- Data Scarcity Demands Stricter Regularization: Smaller datasets (like AFHQ-Cat and FFHQ-10k) require a significantly higher $r$ value (and thus a smaller $m$). This means the discriminator needs tighter constraints to prevent it from overpowering the generator when data is scarce.
- Dynamic Feedback Loop: ABCAS acts like a thermostat for GAN training. By tying the regularization strength directly to the distribution distance, it automatically relaxes or tightens constraints as training conditions change.
Key Results
ABCAS consistently outperformed standard Spectral Normalization across all tested datasets, achieving lower (better) Fréchet Inception Distance (FID) scores:
- AFHQ-Cat: FID dropped from 33.25 (standard SN) to 24.99 with ABCAS ($\beta=4$).
- FFHQ-10k: FID improved from 68.59 to 55.48.
- FFHQ: FID slightly improved from 34.33 to 34.20.
Qualitatively, the images generated via ABCAS displayed significantly fewer structural defects. For instance, in the cat face generation tasks, standard SN often produced unnatural noses or warped facial contours, whereas ABCAS consistently yielded realistic features.
Limitations and Future Work
The authors note that their current implementation relies on a relatively simple heuristic to estimate the distance between real and fake distributions ($\max(C_{real}) - \min(C_{fake})$). While computationally efficient and effective in practice, this metric may not capture full distributional nuances. Future work will focus on exploring more sophisticated and robust metrics to measure distribution distances, which could further refine the adaptive control loop.
Summary updated