Abstract
This article revisits the theoretical foundations of generative modeling, specifically drawing from the frameworks introduced in arXiv:1701.07875 and arXiv:1705.07642. By framing the task as a Minimum Kantorovich Estimator (MKE) problem, we demonstrate that Generative Adversarial Networks (GANs) and Variational Autoencoders (VAEs) are not disparate paradigms but rather dual and primal approximations of the same Optimal Transport objective. This unification provides a rigorous mathematical basis for understanding the trade-offs between stability and sharpness in deep generative models.
Introduction: The Generative Divide
In the mid-2010s, the deep learning community saw the rise of two dominant generative paradigms: Variational Autoencoders (VAEs) and Generative Adversarial Networks (GANs). VAEs were praised for their stable training and principled probabilistic foundations, yet they often produced blurry samples. Conversely, GANs produced high-fidelity, sharp images but were notoriously difficult to train due to instabilities in the adversarial game. This paper addresses the lack of a unified theory by employing Optimal Transport (OT) to show that both models are attempting to solve the same underlying problem.
The Minimum Kantorovich Estimator (MKE)
The core objective of generative modeling is to find a model distribution ($\mu_\theta$) that closely approximates the real data distribution ($\nu$). The authors frame this as a Minimum Kantorovich Estimator (MKE) problem, which seeks to minimize the Wasserstein distance ($W_c$) between these distributions.
The Wasserstein distance represents the minimum cost to "transport" mass from the model distribution to the data distribution. Mathematically, the MKE is defined as: $$\min_\theta W_c(\mu_\theta, \nu)$$ where $\mu_\theta$ is the push-forward of a simple latent distribution $\zeta$ through a generator network $g_\theta$.
Methodology: Dual and Primal Paths
The paper reveals that GANs and VAEs emerge from different mathematical treatments of the Kantorovich problem.
The Dual Path (GANs)
By examining the dual formulation of the Kantorovich problem, the complex coupling between distributions is replaced by a "potential function" $h_\xi$. In the context of deep learning, this potential function acts as a discriminator or "critic." When this potential is parameterized by a neural network, the resulting optimization problem is exactly that of the Wasserstein-GAN (WGAN): $$\min_\theta \max_\xi \int h_\xi(g_\theta(z)) d\zeta(z) + \sum h_\xi^c(y_j)$$
The Primal Path (VAEs)
The primal formulation deals directly with the transport plan. To make this tractable, the authors restrict the transport plan to a parametric form using an encoding map $f_\xi$. Because satisfying transport constraints exactly is difficult, they utilize a relaxed "unbalanced" formulation. This derivation leads directly to the Wasserstein-VAE (WVAE): $$\min_{\theta, \xi} \Delta_\nu(g_\theta \circ f_\xi, \text{Id}_{\mathcal{X}}) + \lambda D(f_\xi \sharp \nu | \zeta)$$
Key Insights and Results
Stability vs. Sharpness
The OT perspective explains the empirical differences between these models. The GAN (dual) gradient relies on the gradient of an optimal potential, which is highly sensitive to the discriminator's quality, leading to training instability. The VAE (primal) approach is more stable as it optimizes a direct reconstruction cost, but the relaxation required for the primal constraints often results in the characteristic "blurriness" of VAE outputs.
The Energy Inequality
The authors establish a fundamental relationship between the energy levels of these estimators: $$E(\theta_{WGAN}) \le E(\theta_{MKE}) \le E(\theta_{WVAE})$$ This implies that WGAN provides a lower bound on the true transport cost, while WVAE provides an upper bound. Theoretically, in the non-parametric limit (infinite network capacity), these bounds converge, and the gap between GANs and VAEs vanishes.
Conclusion
By unifying GANs and VAEs under the umbrella of Optimal Transport, this research clarifies that the choice between these architectures is essentially a choice between dual and primal optimization strategies. While current deep learning implementations optimize "proxy" distances rather than true Wasserstein values, the MKE framework provides a robust theoretical target for future developments in generative modeling and implicit regularization.
Summary updated