Abstract
Recent work in unsupervised representation learning has focused on learning deep directed latent-variable models. Fitting these models by maximizing the marginal likelihood or evidence is typically intractable, thus a common approximation is to maximize the evidence lower bound (ELBO) instead. However, maximum likelihood training (whether exact or approximate) does not necessarily result in a good latent representation, as we demonstrate both theoretically and empirically. In particular, we derive variational lower and upper bounds on the mutual information between the input and the latent variable, and use these bounds to derive a rate-distortion curve that characterizes the tradeoff between compression and reconstruction accuracy. Using this framework, we demonstrate that there is a family of models with identical ELBO, but different quantitative and qualitative characteristics. Our framework also suggests a simple new method to ensure that latent variable models with powerful stochastic decoders do not ignore their latent code.
Introduction
In the field of unsupervised learning, the Variational Autoencoder (VAE) has become a foundational tool. Traditionally, these models are trained by maximizing the Evidence Lower Bound (ELBO), which serves as a proxy for the marginal likelihood of the observed data. However, a persistent challenge in VAE training is the phenomenon of posterior collapse. This occurs when a model, particularly one equipped with a powerful "stochastic" decoder (such as an RNN or PixelCNN), ignores the latent variable $z$ entirely. In such cases, the ELBO may reach a high value, but the learned representation $z$ contains zero information about the input $X$, rendering the model useless for representation learning.
The Rate-Distortion Framework
To address this "broken" ELBO, we propose a framework based on Information Theory, specifically Rate-Distortion (RD) theory. We decompose the ELBO into two competing terms:
- Distortion ($D$): The reconstruction error, measured as the negative log-likelihood. It quantifies the information lost during the encoding-decoding process.
- Rate ($R$): The KL divergence between the encoder's posterior and the prior. It measures the "cost" or number of bits required to transmit the latent code.
By viewing VAE training as a multi-objective optimization on the RD-plane, we can derive fundamental variational bounds on the Mutual Information $I(X;Z)$ between the input and the latent variable: $$H - D \le I(X;Z) \le R$$ where $H$ is the data entropy. This perspective reveals that the standard ELBO (where the weight $\beta=1$) is merely a single point on a convex frontier of possible models.
As shown in the RD-plane diagram, there are feasible and infeasible regions separated by this frontier. Models with identical ELBO scores can exist at different points on this plane, leading to vastly different qualitative behaviors—ranging from "autodecoders" that ignore latents to "semantic encoders" that capture high-level concepts.
Fixing the ELBO: Target Rate Optimization
Our framework suggests that if a decoder is strong enough to model the data distribution independently, we must explicitly constrain the optimization to force the model to utilize the latent code. We propose two primary methods to navigate this tradeoff:
- $\beta$-VAE: Adjusting the weight $\beta \neq 1$ to prioritize either rate or distortion.
- Target Rate: A new objective where we minimize distortion subject to a specific target rate $R^*$. This ensures the model captures a minimum amount of information in its latents.
Empirical Validation: The Toy Model
We evaluated these methods on a synthetic toy model designed with a known "true" mutual information ($I=0.5$ nats). Standard ELBO training failed to recover the underlying generative process, often falling into a state of posterior collapse. However, by employing the Target Rate method, we were able to recover the latent structure perfectly.
In the figure above, the heatmaps for the encoder $e(z|x)$ and decoder $d(x|z)$ demonstrate that the model has learned a meaningful mapping between the input space and the latent space, avoiding the collapse seen in standard VAE implementations.
Key Insights and Results
Beyond the toy model, we applied these principles to complex datasets like Static Binary MNIST and Omniglot. Key findings include:
- Fixing PixelCNN++: On MNIST, a VAE with a PixelCNN++ decoder typically collapses to $R \approx 0$. By setting $\beta < 1$, we forced the model to achieve higher rates, resulting in representations that captured digit identity and style.
- The Prior Bottleneck: Our mapping of Pareto frontiers across various architectures revealed that the primary hurdle to achieving zero distortion is often the prior. Even sophisticated priors struggle to fully close the gap to the theoretical auto-encoding limit.
- KL Annealing Justification: The common heuristic of "KL annealing" is revealed to be a practical way of moving along the Rate-Distortion curve, for which our framework provides a rigorous theoretical basis.
Conclusion
Maximizing the ELBO is not a sufficient objective for learning useful representations. By adopting a Rate-Distortion perspective, researchers can gain finer control over the information flow in latent variable models. We recommend that reporting separate Rate and Distortion values—rather than a single ELBO score—should become the standard practice for evaluating generative models, providing a clearer picture of the representation's quality and the model's efficiency.
Summary updated