Summable Reparameterizations of Wasserstein Critics in the One-Dimensional Setting

Christopher Grimm, Yuhang Song, Michael L. LittmanarXiv 2017 · arXiv

Introduction

Generative Adversarial Networks (GANs) have revolutionized generative modeling but remain notoriously difficult to train due to instabilities in the minimax optimization process. A significant advancement was the introduction of the Wasserstein GAN (wGAN), which utilizes the Earth-Mover’s Distance (EMD) to provide a more meaningful gradient for the generator. However, the effectiveness of wGANs hinges on the "critic" function being 1-Lipschitz—meaning its slope can never exceed 1.

Standard methods like Gradient Penalty (wGAN-GP) attempt to enforce this constraint through heuristics that do not guarantee global optimality or strict adherence to the Lipschitz condition everywhere. This often leads to poor feedback for the generator, resulting in training divergence or "mode collapse." This research addresses these issues by reparameterizing the critic to ensure stability and convergence.

Methods: Summable Reparameterization

To address these limitations in one-dimensional settings, the authors propose replacing standard neural network critics with Summable Reparameterizations. Instead of a deep learning architecture, the critic is defined as a weighted sum of basis functions: $$f(x) = \sum A_{n,m} \xi_{n,m}(x)$$

By selecting basis functions ($\xi$) with bounded derivatives, the 1-Lipschitz constraint can be strictly enforced by simply limiting the sum of the absolute values of the weights ($A$). The authors explore two primary expansions:

  1. Taylor Series: This approach links wGANs to classical moment matching in statistics.
  2. Fourier Series: This uses sinusoidal functions to approximate the optimal critic.

The most significant theoretical contribution is that under this parameterization, the critic's objective function becomes linear and the constraint space becomes convex. This ensures that any local maximum found during optimization is a global maximum, providing a stability guarantee that neural networks lack.

Experimental Setup and Datasets

The proposed Taylor and Fourier critics were evaluated on four distinct 1D distributions, ranging from synthetic shapes to real-world data. These include a Sawtooth distribution, a Discrete distribution with three distinct values, a Gaussian Mixture, and City Populations (log-scaled and normalized data from the Free World Cities Database).

Histograms of the four target 1D distributions: (a) Sawtooth, (b) Gaussian Mixture, (c) Discrete, and (d) City Populations.

Key Results and Stability

The reparameterized critics demonstrated superior performance and stability compared to the standard wGAN-GP baseline. In many cases, the Taylor and Fourier critics achieved EMD values an order of magnitude lower than wGAN-GP.

Summary of Earth-Mover’s Distance (EMD) results across different datasets and models. Lower values indicate better performance.

As shown in the results above, the proposed methods are even competitive with Kernel Density Estimation (KDE), a non-parametric gold standard. Notably, in the Sawtooth dataset, wGAN-GP occasionally failed to converge (EMD ~0.26), whereas the Fourier critic remained highly stable (EMD ~0.01). This confirms that trading the "black-box" flexibility of neural networks for the "optimizability" of series expansions leads to much more reliable generative models in 1D.

Conclusion and Future Work

By moving to a convex optimization framework for the critic, this work provides a theoretical foundation for stable wGAN training in 1D. While the "curse of dimensionality" currently limits this specific series-expansion approach to low-dimensional data—as the number of terms required grows exponentially with dimensions—it opens new avenues for research. Future work may explore sparse expansions to maintain convergence guarantees while scaling to high-dimensional data like images.

Summary updated

Abstract

Generative adversarial networks (GANs) are an exciting alternative to algorithms for solving density estimation problems---using data to assess how likely samples are to be drawn from the same distribution. Instead of explicitly computing these probabilities, GANs learn a generator that can match the given probabilistic source. This paper looks particularly at this matching capability in the context of problems with one-dimensional outputs. We identify a class of function decompositions with properties that make them well suited to the critic role in a leading approach to GANs known as Wasserstein GANs. We show that Taylor and Fourier series decompositions belong to our class, provide examples of these critics outperforming standard GAN approaches, and suggest how they can be scaled to higher dimensional problems in the future.
BibTeX
@misc{grimm2017summable,
  author = {Christopher Grimm and Yuhang Song and Michael L. Littman},
  title = {{Summable Reparameterizations of Wasserstein Critics in the One-Dimensional Setting}},
  year = {2017},
  eprint = {1709.06533},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1709.06533},
}
Figure