An Information-Theoretic Analysis of Deep Latent-Variable Models

Alexander A. Alemi, Ben Poole, Ian Fischer, Joshua V. Dillon, Rif A. Saurous, Kevin Murphy

Introduction: The ELBO and Posterior Collapse

In unsupervised learning, the primary objective is to learn useful latent representations of data. The standard approach utilizes the Variational Autoencoder (VAE), which is trained by maximizing the Evidence Lower Bound (ELBO). However, a persistent challenge in this field is posterior collapse. This phenomenon occurs when a decoder is sufficiently "powerful" (e.g., an autoregressive model like PixelCNN) to model the data distribution independently, leading it to ignore the latent code $z$ entirely.

From the perspective of the ELBO, a model that ignores the latent code but achieves perfect reconstruction is mathematically equivalent to one that utilizes the latent code effectively. Because the ELBO cannot distinguish between these qualitatively different behaviors, it is often insufficient for ensuring meaningful representation learning. This paper provides an information-theoretic framework to diagnose this trade-off and proposes methods to enforce information flow into the latent variables.

Theoretical Framework: Rate-Distortion Theory

The authors analyze VAEs through the lens of Rate-Distortion (RD) theory. They decompose the learning objective into three fundamental components:

  1. Entropy ($H$): The inherent complexity of the data distribution.
  2. Distortion ($D$): The reconstruction error, typically measured as negative log-likelihood.
  3. Rate ($R$): The amount of information stored in the latent code, measured as the KL divergence between the encoder and the prior.

A key theoretical contribution is the proof that the mutual information $I(X;Z)$ between the input and the latent variable is bounded by these values: $H - D \leq I(X;Z) \leq R$. This allows researchers to visualize model performance on an RD-plane (Rate vs. Distortion).

The Rate-Distortion plane illustrating the trade-off between information rate (R) and reconstruction distortion (D). The $\beta=1$ line represents the standard ELBO objective, while varying $\beta$ allows navigation along the Pareto frontier.

To explore the capabilities of a model architecture, the authors utilize the $\beta$-VAE objective ($D + \beta R$). By varying the hyperparameter $\beta$, they can trace the "Pareto frontier" of the model.

  • $\beta > 1$: Encourages higher compression (lower rate), often leading to more disentangled but less detailed representations.
  • $\beta < 1$: Prioritizes reconstruction, forcing the model to use the latent channel even when the decoder is powerful.

Crucially, the authors argue that the standard ELBO (where $\beta=1$) is merely a single point on this curve, not an absolute goal. If a model's architecture is imbalanced—such as a weak encoder paired with a strong decoder—the $\beta=1$ point may naturally fall into a region of zero rate, causing posterior collapse. To remedy this, they propose a target rate constraint ($R^*$), which optimizes the model to capture a specific, desired amount of information.

Experimental Results

The framework was evaluated across synthetic and real-world datasets, including Static Binary MNIST and Omniglot.

The Toy Model

Using a synthetic dataset with a known generative process, the authors demonstrated that a standard VAE often fails to recover the true latent structure, gravitating toward a zero-rate solution. However, by applying the target rate constraint, the model was forced to utilize the latent channel, successfully recovering the underlying mutual information of the source.

Comparison of (a) an optimal hand-constructed model, (b) a standard VAE showing posterior collapse where the latent space is ignored, and (c) the proposed Target Rate method which successfully recovers the latent structure and conditional distributions.

Semantic vs. Syntactic Encoding

On MNIST, the authors identified distinct regimes on the RD curve. At lower rates, models tend to learn semantic features, such as digit class. As the rate increases, the models begin to capture syntactic features, such as specific stroke widths or textures. By using $\beta < 1$, they successfully forced VAEs with PixelCNN decoders to stop ignoring their latents, resulting in "semantic encoders" that capture the essence of the data while leaving low-level details to the decoder.

Conclusion and Future Work

The Rate-Distortion framework makes the behavior of deep latent-variable models visible and measurable. It reveals that posterior collapse is often a consequence of the objective function landing on a sub-optimal point of the RD curve. While the authors demonstrate that information constraints can force models to use latent codes, they note that current models still struggle to reach the theoretical "auto-encoding limit" (zero distortion at the data's entropy). This suggests that future research should focus on more sophisticated, learnable priors to allow models to reach higher information rates without sacrificing reconstruction quality.

Summary updated

Abstract

Recent work in unsupervised representation learning has focused on learning deep directed latent-variable models. Fitting these models by maximizing the marginal likelihood or evidence is typically intractable, thus a common approximation is to maximize the evidence lower bound (ELBO) instead. However, maximum likelihood training (whether exact or approximate) does not necessarily result in a good latent representation, as we demonstrate both theoretically and empirically. In particular, we derive variational lower and upper bounds on the mutual information between the input and the latent variable, and use these bounds to derive a rate-distortion curve that characterizes the tradeoff between compression and reconstruction accuracy. Using this framework, we demonstrate that there is a family of models with identical ELBO, but different quantitative and qualitative characteristics. Our framework also suggests a simple new method to ensure that latent variable models with powerful stochastic decoders do not ignore their latent code.
BibTeX
@misc{alemi2017information,
  author = {Alexander A. Alemi and Ben Poole and Ian Fischer and Joshua V. Dillon and Rif A. Saurous and Kevin Murphy},
  title = {{An Information-Theoretic Analysis of Deep Latent-Variable Models}},
  year = {2017},
  eprint = {1711.00464},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1711.00464},
}
Figure