Abstract
This paper introduces a unifying mathematical framework for the diverse "zoo" of latent variable generative models—including Variational Autoencoders (VAEs), Generative Adversarial Networks (GANs), and their variants—through the lens of Lagrangian duality. By reframing model training as a constrained optimization problem, the authors demonstrate that existing objectives are specific instances of a dual function with fixed multipliers. Building on this, they propose the Lagrangian VAE (LagVAE), which treats these multipliers as learnable parameters to automatically balance the trade-off between data reconstruction and representation learning, achieving Pareto-optimal performance.
Introduction
In recent years, the machine learning community has witnessed an explosion of training objectives for generative models, such as $\beta$-VAE, InfoGAN, ALI, and CycleGAN. While these models all aim to learn a mapping between visible data ($x$) and latent variables ($z$), they have largely been treated as a disconnected collection of distinct loss functions and hyperparameters. This fragmentation leads to two primary challenges: a lack of theoretical unification and the difficulty of manual hyperparameter tuning (e.g., selecting the optimal $\beta$ in a $\beta$-VAE).
As shown in Table 1, the authors categorize these models based on their likelihood dependencies and objective functions, providing a roadmap for understanding how seemingly disparate methods relate to one another under a single mathematical umbrella.
A Unified Lagrangian Framework
The authors propose that most generative models are solving a common Primal Optimization Problem:
- Objective: Maximize or minimize the Mutual Information $I(x; z)$ between the data and the latent representations.
- Constraints: Enforce "consistency" between the generative process ($z \to x$) and the inference process ($x \to z$). This requires the joint distribution $p(z)p(x|z)$ to match $q(x)q(z|x)$.
By applying Lagrangian multipliers ($\lambda$) to these consistency constraints, the authors show that existing models are essentially specific instances of this dual function where the multipliers are kept fixed. This perspective reframes generative modeling as the enforcement of symmetry between the encoder and the decoder.
The Lagrangian VAE (LagVAE)
To address the limitations of fixed hyperparameters, the authors introduce LagVAE. Instead of manually picking a weight and hoping for optimal results, LagVAE treats the multipliers as learnable parameters within an iterative min-max optimization process:
- Minimize the objective with respect to model parameters ($\theta$).
- Maximize with respect to the Lagrange multipliers ($\lambda$).
This approach allows users to specify a target "budget" for reconstruction error (a slack variable $\epsilon$), and the algorithm automatically discovers the maximum mutual information possible within that constraint.
Theoretical Insights
- The Closure Theorem: The authors prove that nearly all existing KL-based Lagrangian objectives are linear combinations of a few "base" models (like VAE and InfoGAN), suggesting that the mathematical space of simple linear objectives is largely explored.
- Strong Duality: The paper demonstrates that "strong duality" holds in distribution space. This ensures that solving the dual problem (the training objective) is mathematically guaranteed to solve the original primal problem of interest.
- Dynamic Multipliers: The framework argues that weights like $\beta$ should not be static hyperparameters but dynamic variables that evolve during training to satisfy specific constraint targets.
Experimental Evaluation
The framework was validated primarily on the MNIST dataset to allow for precise measurement of Mutual Information and the Evidence Lower Bound (ELBO).
Key Results
- Pareto Optimality: LagVAE was found to Pareto-dominate InfoVAE. For any specific level of reconstruction quality, LagVAE identifies a representation with higher mutual information than models using manually tuned fixed weights.
- Constraint Reliability: Unlike standard VAEs, where final reconstruction error is a byproduct of the chosen $\beta$, LagVAE reliably converges to the user-defined consistency target ($\epsilon$).
- Optimization Stability: Despite the non-convex nature of deep neural networks, the min-max dual optimization proved stable and converged effectively across experiments.
Discussion and Future Work
While the current work focuses on Mutual Information as the primary preference function, future research could explore other objectives, such as encouraging specific topological properties in latent space. Additionally, while the theory applies to various divergences (KL, MMD, Wasserstein), expanding experiments to high-resolution datasets like CelebA or ImageNet remains a necessary step to test the framework's scalability. Finally, extending the proof for strong duality from discrete to continuous spaces remains an open theoretical challenge.
Summary updated