TL;DR
Dynamic Sparse Training (DST) cuts the massive compute needed to train neural networks by keeping them sparse from scratch, but applying it to Generative Adversarial Networks (GANs) frequently fails due to delicate adversarial imbalances. This paper introduces the Balance Ratio (BR)—a metric that tracks whether the generator or discriminator is dominating—and uses it to develop Double Dynamic Sparse Training (DDST), an approach that dynamically tunes both model density and sparse connectivity on the fly to stabilize training and drastically cut training FLOPs.
What Problem Does This Paper Address?
Training state-of-the-art GANs requires massive computational resources (for instance, training BigGAN on 8 V100 GPUs takes two weeks). While Dynamic Sparse Training (DST) has proven effective in supervised learning—allowing networks to maintain a fixed parameter budget throughout training while dynamic pruning and regrowth find optimal sparse structures—translating DST to GANs is notoriously tricky.
GAN stability relies heavily on a delicate balance: an overly powerful discriminator causes the generator's gradients to vanish or overfit, while an overly weak discriminator triggers mode collapse. Prior sparse GAN methods either relied on static sub-networks or only updated the generator's sparsity mask (Single Dynamic Sparse Training, or SDST), leaving the discriminator density statically chosen by guesswork. Picking the wrong discriminator capacity often leads to total training failure or massive waste of computational budget.
Methods
The authors introduce a measurement and a two-sided dynamic framework:
-
Balance Ratio (BR): Quantifies generator improvement per training step relative to the discriminator's scale: $$\text{BR} = \frac{D(G^{\text{post}}(z)) - D(G^{\text{pre}}(z))}{D(x_r) - D(G^{\text{pre}}(z))}$$ Here, $G^{\text{pre}}$ and $G^{\text{post}}$ are generator states before and after a gradient step. If BR is too low ($\le 0.3$), the generator is failing to learn against an overly strong discriminator. If BR is too high or oscillating above $1.0$, the discriminator is too weak to provide a meaningful training signal. A healthy equilibrium generally stays within $[0.3, 0.7]$.
-
Dynamic Density Adjustment (DA): A controller that tracks time-averaged BR. If BR drops below a lower bound $B_-$, the discriminator density is pruned by $\Delta d$; if it rises above $B_+$, discriminator density is expanded via gradient-based or random weight growth.
- DDST Variants:
- Relaxed DDST (R-DDST): The generator runs DST (e.g., using RigL or SET) while the discriminator's density is adaptively adjusted with DA up to 100% capacity.
- Strict DDST (S-DDST): Enforces strict memory and compute caps by bounding discriminator density (e.g., $d_D \le 50\%$). Training is split into two phases: (1) Density exploration using DA to find the optimal discriminator density $d_D^*$, followed by (2) Parameter exploration, where both networks update their sparse topologies via DST under fixed density budgets.
Datasets
The authors evaluated their methods on standard generative benchmarks without introducing new datasets:
- CIFAR-10 ($32 \times 32$ resolution), evaluated on SNGAN (ResNet-32) and BigGAN.
- STL-10 (unlabeled split resized to $48 \times 48$), evaluated on SNGAN (ResNet-48).
Key Insights and Ideas
- Overly weak discriminators are fatal; overly strong ones are merely inefficient: When discriminator density is starved ($d_D = 10\%$), BR spikes above $1.0$ and enters chaotic oscillations, closely preceding severe FID degradation. In contrast, an over-parameterized discriminator causes minor quality drop-offs, making under-parameterization the far deadlier failure mode.
- Generator DST needs a matching sparring partner: SDST injects "in-time over-parameterization" into the generator. If the discriminator is too sparse, this extra generator capacity overwhelms it.
- Gradient-based growth scales better: While random regrowth (SET) suffices on CIFAR-10, gradient-guided growth (RigL) significantly outperforms SET on larger networks and datasets (like STL-10) by allocating discriminator capacity where it matters most.
Key Results
- Significant FLOPs Reduction with BigGAN on CIFAR-10:
- At 10% generator density, R-DDST (RigL) achieved an FID of 9.58 while using only 9.77% of dense training FLOPs.
- In comparison, standard SDST with a dense discriminator achieved a worse FID of 10.79 while burning 84.04% of dense FLOPs.
- Competitive Generative Quality: On CIFAR-10 SNGAN at 50% generator density, R-DDST reached 10.97 FID, closely approaching the dense baseline (10.74 FID) and matching or outperforming static sparse setups across all densities ($10\% - 50\%$).
- Strictly Sparse Performance: Even with the discriminator strictly capped at $50\%$ density, S-DDST (RigL) scored 14.13 FID on CIFAR-10 at 10% generator density, comfortably outperforming the static-strong baseline (22.35) and DST-balance (24.56).
Limitations and Future Work
- Computational Overhead of Tracking BR: Computing BR requires extra discriminator forward passes on generated samples before and after the update. While computing it every few hundred iterations mitigates the cost, doing it continuously adds overhead.
- Benchmark Scale: The framework was tested on moderate-resolution datasets ($32\times32$ and $48\times48$). Validating DDST on high-resolution setups (e.g., ImageNet at $256\times256$ or StyleGAN architectures) remains an open challenge.
- Heuristic Phase Scheduling in S-DDST: S-DDST splits the training budget into density and parameter exploration evenly at 50% by default; an adaptive trigger for transitioning phases could improve performance and efficiency.
Summary updated