Lipschitz-Constrained Discriminators in Generative Modeling: Asymptotic Behavior and Support Mismatch

Eduardo Pinto, Panagiota BirmpaarXiv 2026 · arXiv

TL;DR

This paper provides a rigorous theoretical bridge between geometry-aware Wasserstein distances and density-matching KL divergences. By studying Lipschitz-constrained discriminators, the authors prove that as the constraint is relaxed, these "hybrid" discriminators converge to the true KL discriminator—even in the tricky "support mismatch" regime where the model and data distributions don't overlap.

What Problem Does This Paper Address?

In generative modeling (like training GANs), we want a model distribution ($Q$) to match a target data distribution ($P$). Traditionally, we use measures like the Kullback-Leibler (KL) divergence to quantify the difference. However, KL divergence has a major weakness: if the supports of $P$ and $Q$ don't overlap (which is almost always the case at the start of training), the KL divergence becomes infinite and provides no useful "gradient" or direction for the model to improve.

Optimal Transport (like the Wasserstein-1 distance) solves this by being "geometry-aware"—it cares about how far points are from each other, not just if they overlap. Researchers often use $(f, \Gamma)$-divergences to interpolate between these two worlds. While we knew the values of these hybrid divergences converged to KL as the Lipschitz constraint ($L$) increased, we didn't have a clear picture of how the optimal discriminators (the functions that actually guide the learning) behaved in that limit. This paper fills that gap.

Methods

The authors analyze the asymptotic behavior of optimal discriminators $\phi^*_L$ as the Lipschitz constant $L \to \infty$. Their approach relies on:

  • Variational Representations: They use the dual form of the KL divergence, restricted to the class of $L$-Lipschitz functions.
  • $\Gamma$-convergence: This is a specialized mathematical framework from the calculus of variations used to study the limits of optimization problems. It allows the authors to show that the "path" of optimal functions $\phi^*_L$ leads directly to the unconstrained KL discriminator.
  • Extended Function Spaces: To handle cases where distributions don't overlap, they allow the discriminator to take values of $\pm\infty$, creating a rigorous way to talk about "infinite" divergence in a discrete setting.

Datasets

The paper is primarily theoretical and uses synthetic, discrete probability measures to validate the proofs:

  • 1D Empirical Measures: Simple discrete points in $\mathbb{R}$ used to visualize how the discriminator's shape changes as $L$ increases.
  • 4D Disjoint Distributions: 200 support points sampled from Gaussian distributions in $\mathbb{R}^4$ to test the theory in higher dimensions.
  • No real-world benchmarks (like ImageNet) were used, as the focus is on fundamental mathematical convergence rather than engineering a new state-of-the-art model.

Key Insights and Ideas

  • The "Continuation" Strategy: The results suggest a principled training schedule. Start with a small $L$ (strong Lipschitz constraint) to get a smooth, geometry-aware signal when the model is far from the data. As the model gets closer, gradually increase $L$ to transition into sharp "density matching" via the KL regime.
  • Predictable Explosion: When supports are disjoint, the KL discriminator should be infinite. The authors show that the Lipschitz-constrained versions don't just behave randomly; they converge to a function that is $+\infty$ on the model support and bounded on the data support.
  • Geometry vs. Information: The parameter $L$ acts as a dial. Low $L$ enforces geometric regularity (Wasserstein-like), while high $L$ relaxes that regularity to focus on the information-theoretic density ratio (KL-like).

Key Results

  • Theorem 4.1 (The Ideal Case): If the density ratio is bounded and the distributions overlap, the Lipschitz discriminators converge $P$-almost surely to the unique KL discriminator.
  • The Support Mismatch Result: In discrete settings where $Q$ and $P$ have disjoint supports, the authors prove that the discriminator values on the model's support tend toward $+\infty$, while remaining bounded on the data's support.
  • Numerical Confirmation: In their experiments, the hybrid divergence value $D_{KL}^L$ rapidly approached the true KL value as $L$ increased (e.g., moving from $L=1$ to $L=10$ captured most of the divergence value).
  • Visual Evolution: Plots show that as $L$ increases, the discriminator transforms from a smooth, slanted line (geometry-aware) into a sharp, step-like function (density-ratio aware).

Limitations and Future Work

  • Computational Scaling: The exact solution for the Lipschitz-constrained problem involves $O(n^2)$ constraints for $n$ data points. This is feasible for the paper's examples ($n=200$) but would require stochastic approximations or neural network parameterization for large-scale applications.
  • Adaptive Schedules: While the paper motivates increasing $L$ during training, it doesn't provide an automated "rule" for how fast $L$ should grow to optimize training speed or stability.
  • Compactness Assumption: The current proofs assume distributions are concentrated on a compact set $K \subset \mathbb{R}^d$. Extending this to non-compact spaces (like the entire real line) remains for future work.

Summary updated

Abstract

We study the asymptotic behavior of optimal discriminators associated to $L-$Lipschitz regularized KL-divergences of two probability distributions $Q$ and $P$ that interpolate between the rescaled Wasserstein$-1$ metric by some parameter $L>0$ and the KL divergence. The parameter $L$ plays the role of a Lipschitz constraint and is viewed as a regularization parameter: small values of $L$ enforce a more geometry-aware discriminator, while as $L$ increases the geometric regularization is progressively relaxed. It is a known result [3] that such divergences converge to the KL divergence as $L\to\infty$. In this paper, we characterize the asymptotic behavior of the optimal discriminators as $L\to\infty$ in different regimes: Firstly, under $Q\ll P$ and the Radon-Nikodym derivative $\frac{dQ}{dP}$ is bounded from above and bounded away from zero. Secondly, when $Q$ and $P$ are discrete distributions satisfying $Q\ll P$, and finally when $Q$ and $P$ are non absolutely continuous discrete distributions. This result motivates training strategies which involve discriminators, such as adversarial generative models. In practice, the data distribution is often represented by a discrete empirical measure, while, especially at the early stages of the training, the model distribution may have little or no overlap with the empirical data distribution.
BibTeX
@misc{pinto2026lipschitz,
  author = {Eduardo Pinto and Panagiota Birmpa},
  title = {{Lipschitz-Constrained Discriminators in Generative Modeling: Asymptotic Behavior and Support Mismatch}},
  year = {2026},
  eprint = {2611.18394},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.18394},
}
Figure