Double Dynamic Sparse Training for GANs

Yite Wang, Jing Wu, Naira Hovakimyan, Ruoyu Sun

Abstract

The past decade has witnessed a drastic increase in modern deep neural networks (DNNs) size, especially for generative adversarial networks (GANs). Since GANs usually suffer from high computational complexity, researchers have shown an increased interest in applying pruning methods to reduce the training and inference costs of GANs. Among different pruning methods invented for supervised learning, dynamic sparse training (DST) has gained increasing attention recently as it enjoys excellent training efficiency with comparable performance to post-hoc pruning. Hence, applying DST on GANs, where we train a sparse GAN with a fixed parameter count throughout training, seems to be a good candidate for reducing GAN training costs. However, a few challenges, including the degrading training instability, emerge due to the adversarial nature of GANs. Hence, we introduce a quantity called balance ratio (BR) to quantify the balance of the generator and the discriminator. We conduct a series of experiments to show the importance of BR in understanding sparse GAN training. Building upon single dynamic sparse training (SDST), where only the generator is adjusted during training, we propose double dynamic sparse training (DDST) to control the BR during GAN training. Empirically, DDST automatically determines the density of the discriminator and greatly boosts the performance of sparse GANs on multiple datasets.

Introduction

Training Generative Adversarial Networks (GANs) is notoriously resource-heavy. While Dynamic Sparse Training (DST)—which prunes and regrows neural connections during the training process—has been successful in supervised learning, it struggles with GANs. In a GAN, the Generator ($G$) and Discriminator ($D$) are locked in an arms race. If $D$ is too strong, $G$ receives useless gradients and stops learning; if $D$ is too weak, $G$ can "cheat," leading to mode collapse. When sparsity is introduced, this balance becomes even more fragile. Naively applying DST to a GAN usually results in a lopsided competition where one component dominates, leading to poor image quality or total training failure.

Methods: The Balance Ratio and ADAPT

To stabilize sparse GAN training, this research introduces two major contributions. The first is the Balance Ratio (BR), a new "health metric" for GANs. It measures the ratio between how much the generator improves in a single step ($\alpha$) versus the current performance gap between the discriminator and the generator ($\beta$).

Illustration of the Balance Ratio (BR) calculation, showing the relationship between generator improvement ($\alpha$) and the discriminator-generator gap ($\beta$).
  • Low BR: The discriminator is too strong; the generator cannot keep up.
  • High BR: The discriminator is too weak; it is not providing enough challenge.

Building on this, the ADAPT (bAlanced DynAmic sParse Training) algorithm—an implementation of Double Dynamic Sparse Training (DDST)—uses the BR as a feedback loop. During training, ADAPT monitors the BR. If the ratio drifts outside a target "Goldilocks zone," the algorithm automatically adjusts the discriminator density. ADAPT-relax allows the discriminator's density to increase up to 100% if needed to challenge a strong generator, while ADAPT-strict keeps the discriminator sparse but uses DST to regrow more effective connections.

Experimental Results and Insights

The researchers evaluated their methods using SNGAN and BigGAN architectures across CIFAR-10, STL-10, and TinyImageNet. A key insight is the existence of a "Goldilocks" discriminator: neither maximum nor minimum density is ideal for a given generator density.

FID scores across different discriminator densities ($d_D$) for various fixed generator densities ($d_G$). The plots demonstrate that an optimal discriminator density exists for each generator density level.

Key findings include:

  • Superior Quality: ADAPT_relax achieved the best FID (Fréchet Inception Distance) in 13 out of 16 experimental cases. For example, on SNGAN (CIFAR-10), it improved FID by significant margins over static sparse baselines.
  • Efficiency Gains: On BigGAN (CIFAR-10), ADAPT achieved better results than the next-best sparse method while using only ~10% of the training FLOPs (compared to ~84% for the baseline).
  • Stability: Unlike standard DST, ADAPT's dynamic density adjustment kept training stable across all tested sparsity levels, preventing the training collapse often seen when the discriminator is too weak.

Conclusion

This work demonstrates that maintaining the competitive balance between the generator and discriminator is critical for successful sparse GAN training. By introducing the Balance Ratio as a diagnostic tool and the ADAPT algorithm to maintain equilibrium, the authors achieve state-of-the-art results with a fraction of the computational cost. Future work will investigate how to determine the ideal BR range automatically and test these principles on larger architectures like StyleGAN3.

Summary updated

Abstract

The past decade has witnessed a drastic increase in modern deep neural networks (DNNs) size, especially for generative adversarial networks (GANs). Since GANs usually suffer from high computational complexity, researchers have shown an increased interest in applying pruning methods to reduce the training and inference costs of GANs. Among different pruning methods invented for supervised learning, dynamic sparse training (DST) has gained increasing attention recently as it enjoys excellent training efficiency with comparable performance to post-hoc pruning. Hence, applying DST on GANs, where we train a sparse GAN with a fixed parameter count throughout training, seems to be a good candidate for reducing GAN training costs. However, a few challenges, including the degrading training instability, emerge due to the adversarial nature of GANs. Hence, we introduce a quantity called balance ratio (BR) to quantify the balance of the generator and the discriminator. We conduct a series of experiments to show the importance of BR in understanding sparse GAN training. Building upon single dynamic sparse training (SDST), where only the generator is adjusted during training, we propose double dynamic sparse training (DDST) to control the BR during GAN training. Empirically, DDST automatically determines the density of the discriminator and greatly boosts the performance of sparse GANs on multiple datasets.
BibTeX
@inproceedings{wang2023double,
  author = {Yite Wang and Jing Wu and Naira Hovakimyan and Ruoyu Sun},
  title = {{Double Dynamic Sparse Training for GANs}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=wmMUAg_l4Qk},
}
Figure