TL;DR
This paper provides a rigorous theoretical bridge between geometry-aware Wasserstein distances and density-matching KL divergences. By studying Lipschitz-constrained discriminators, the authors prove that as the constraint is relaxed, these "hybrid" discriminators converge to the true KL discriminator—even in the tricky "support mismatch" regime where the model and data distributions don't overlap.
What Problem Does This Paper Address?
In generative modeling (like training GANs), we want a model distribution ($Q$) to match a target data distribution ($P$). Traditionally, we use measures like the Kullback-Leibler (KL) divergence to quantify the difference. However, KL divergence has a major weakness: if the supports of $P$ and $Q$ don't overlap (which is almost always the case at the start of training), the KL divergence becomes infinite and provides no useful "gradient" or direction for the model to improve.
Optimal Transport (like the Wasserstein-1 distance) solves this by being "geometry-aware"—it cares about how far points are from each other, not just if they overlap. Researchers often use $(f, \Gamma)$-divergences to interpolate between these two worlds. While we knew the values of these hybrid divergences converged to KL as the Lipschitz constraint ($L$) increased, we didn't have a clear picture of how the optimal discriminators (the functions that actually guide the learning) behaved in that limit. This paper fills that gap.
Methods
The authors analyze the asymptotic behavior of optimal discriminators $\phi^*_L$ as the Lipschitz constant $L \to \infty$. Their approach relies on:
- Variational Representations: They use the dual form of the KL divergence, restricted to the class of $L$-Lipschitz functions.
- $\Gamma$-convergence: This is a specialized mathematical framework from the calculus of variations used to study the limits of optimization problems. It allows the authors to show that the "path" of optimal functions $\phi^*_L$ leads directly to the unconstrained KL discriminator.
- Extended Function Spaces: To handle cases where distributions don't overlap, they allow the discriminator to take values of $\pm\infty$, creating a rigorous way to talk about "infinite" divergence in a discrete setting.
Datasets
The paper is primarily theoretical and uses synthetic, discrete probability measures to validate the proofs:
- 1D Empirical Measures: Simple discrete points in $\mathbb{R}$ used to visualize how the discriminator's shape changes as $L$ increases.
- 4D Disjoint Distributions: 200 support points sampled from Gaussian distributions in $\mathbb{R}^4$ to test the theory in higher dimensions.
- No real-world benchmarks (like ImageNet) were used, as the focus is on fundamental mathematical convergence rather than engineering a new state-of-the-art model.
Key Insights and Ideas
- The "Continuation" Strategy: The results suggest a principled training schedule. Start with a small $L$ (strong Lipschitz constraint) to get a smooth, geometry-aware signal when the model is far from the data. As the model gets closer, gradually increase $L$ to transition into sharp "density matching" via the KL regime.
- Predictable Explosion: When supports are disjoint, the KL discriminator should be infinite. The authors show that the Lipschitz-constrained versions don't just behave randomly; they converge to a function that is $+\infty$ on the model support and bounded on the data support.
- Geometry vs. Information: The parameter $L$ acts as a dial. Low $L$ enforces geometric regularity (Wasserstein-like), while high $L$ relaxes that regularity to focus on the information-theoretic density ratio (KL-like).
Key Results
- Theorem 4.1 (The Ideal Case): If the density ratio is bounded and the distributions overlap, the Lipschitz discriminators converge $P$-almost surely to the unique KL discriminator.
- The Support Mismatch Result: In discrete settings where $Q$ and $P$ have disjoint supports, the authors prove that the discriminator values on the model's support tend toward $+\infty$, while remaining bounded on the data's support.
- Numerical Confirmation: In their experiments, the hybrid divergence value $D_{KL}^L$ rapidly approached the true KL value as $L$ increased (e.g., moving from $L=1$ to $L=10$ captured most of the divergence value).
- Visual Evolution: Plots show that as $L$ increases, the discriminator transforms from a smooth, slanted line (geometry-aware) into a sharp, step-like function (density-ratio aware).
Limitations and Future Work
- Computational Scaling: The exact solution for the Lipschitz-constrained problem involves $O(n^2)$ constraints for $n$ data points. This is feasible for the paper's examples ($n=200$) but would require stochastic approximations or neural network parameterization for large-scale applications.
- Adaptive Schedules: While the paper motivates increasing $L$ during training, it doesn't provide an automated "rule" for how fast $L$ should grow to optimize training speed or stability.
- Compactness Assumption: The current proofs assume distributions are concentrated on a compact set $K \subset \mathbb{R}^d$. Extending this to non-compact spaces (like the entire real line) remains for future work.
Summary updated