Introduction
Generative Adversarial Networks (GANs) have revolutionized generative modeling but remain notoriously difficult to train due to instabilities in the minimax optimization process. A significant advancement was the introduction of the Wasserstein GAN (wGAN), which utilizes the Earth-Mover’s Distance (EMD) to provide a more meaningful gradient for the generator. However, the effectiveness of wGANs hinges on the "critic" function being 1-Lipschitz—meaning its slope can never exceed 1.
Standard methods like Gradient Penalty (wGAN-GP) attempt to enforce this constraint through heuristics that do not guarantee global optimality or strict adherence to the Lipschitz condition everywhere. This often leads to poor feedback for the generator, resulting in training divergence or "mode collapse." This research addresses these issues by reparameterizing the critic to ensure stability and convergence.
Methods: Summable Reparameterization
To address these limitations in one-dimensional settings, the authors propose replacing standard neural network critics with Summable Reparameterizations. Instead of a deep learning architecture, the critic is defined as a weighted sum of basis functions: $$f(x) = \sum A_{n,m} \xi_{n,m}(x)$$
By selecting basis functions ($\xi$) with bounded derivatives, the 1-Lipschitz constraint can be strictly enforced by simply limiting the sum of the absolute values of the weights ($A$). The authors explore two primary expansions:
- Taylor Series: This approach links wGANs to classical moment matching in statistics.
- Fourier Series: This uses sinusoidal functions to approximate the optimal critic.
The most significant theoretical contribution is that under this parameterization, the critic's objective function becomes linear and the constraint space becomes convex. This ensures that any local maximum found during optimization is a global maximum, providing a stability guarantee that neural networks lack.
Experimental Setup and Datasets
The proposed Taylor and Fourier critics were evaluated on four distinct 1D distributions, ranging from synthetic shapes to real-world data. These include a Sawtooth distribution, a Discrete distribution with three distinct values, a Gaussian Mixture, and City Populations (log-scaled and normalized data from the Free World Cities Database).
Key Results and Stability
The reparameterized critics demonstrated superior performance and stability compared to the standard wGAN-GP baseline. In many cases, the Taylor and Fourier critics achieved EMD values an order of magnitude lower than wGAN-GP.
As shown in the results above, the proposed methods are even competitive with Kernel Density Estimation (KDE), a non-parametric gold standard. Notably, in the Sawtooth dataset, wGAN-GP occasionally failed to converge (EMD ~0.26), whereas the Fourier critic remained highly stable (EMD ~0.01). This confirms that trading the "black-box" flexibility of neural networks for the "optimizability" of series expansions leads to much more reliable generative models in 1D.
Conclusion and Future Work
By moving to a convex optimization framework for the critic, this work provides a theoretical foundation for stable wGAN training in 1D. While the "curse of dimensionality" currently limits this specific series-expansion approach to low-dimensional data—as the number of terms required grows exponentially with dimensions—it opens new avenues for research. Future work may explore sparse expansions to maintain convergence guarantees while scaling to high-dimensional data like images.
Summary updated