Information Theoretic Constraints for Useful Latent Representations

This direction is merged into another direction: posterior collapse. Its last version stays readable and citable here.

Version 1 of 116 papers24 conceptsfrozen 2026-10-04

In the standard recipe for training Variational Autoencoders (VAEs), we maximize the Evidence Lower Bound (ELBO). On paper, this objective balances reconstruction accuracy with a regularizer that keeps the latent space organized. In practice, however, the ELBO is often "broken."

As researchers like Alexander A Alemi and Ben Poole have demonstrated in Fixing a Broken ELBO, a high ELBO score does not guarantee a useful representation. A model can achieve near-perfect likelihood by relying on a powerful decoder while completely ignoring the latent variables—a phenomenon known as posterior collapse. This research direction focuses on moving beyond simple likelihood to ensure that latent spaces are informative, identifiable, and structured for downstream utility.

The Failure of Reconstruction Likelihood

The fundamental problem is that the ELBO is a scalar proxy for data likelihood, but likelihood is not the same as representation quality. Michael Tschannen and Mario Lučić argue in Recent Advances in Autoencoder-Based Representation Learning that models with identical likelihood scores can capture entirely different features.

If a decoder is flexible enough (like a deep GRU or a PixelCNN), it can learn to model the data distribution $p(x)$ independently of the latent code $z$. In these cases, the model "cheats" the ELBO: it minimizes the KL-divergence by setting the posterior to the prior, effectively shutting off the information flow from the encoder. To the ELBO, this is a valid local optimum, but to a researcher wanting to use $z$ for classification or optimization, the model is useless.

Diagnosing with Rate-Distortion Theory

To fix the ELBO, we first need to understand the trade-offs it makes. Alexander A Alemi and colleagues applied Rate-Distortion theory to VAEs in An information-theoretic analysis of deep latent-variable models. They decompose the objective into:

  1. Rate ($R$): The amount of information the latent code carries about the input.
  2. Distortion ($D$): The reconstruction error.

By visualizing models on a Rate-Distortion curve, they showed that posterior collapse occurs when the model finds a point where $R=0$. To prevent this, we can explicitly target specific points on the curve, forcing the model to maintain a minimum "rate" of information flow, regardless of how powerful the decoder is.

Maximizing Mutual Information

If the goal is to make $z$ a meaningful summary of $x$, we should explicitly maximize the Mutual Information (MI) between them. Standard VAEs only maximize MI implicitly.

Jiaming Song and Stefano Ermon introduced InfoVAE (InfoVAE: Balancing Learning and Inference in Variational Autoencoders) to address this. By adding an explicit MI term to the objective, they force the model to prefer latent codes that are highly predictive of the input. Similarly, Andriy Serdega proposed the VMI-VAE framework (VMI-VAE: Variational Mutual Information Maximization Framework for VAE With Discrete and Continuous Priors), which uses a variational lower bound on MI as a regularizer. This ensures that even with discrete priors, the latent code captures the most salient features of the data rather than being ignored by the decoder.

Shaping Latent Space for Utility

Beyond just ensuring the latents are "informative," we often need them to be "useful" for specific tasks. Unsupervised reconstruction doesn't know that you might want to use the latent space for Bayesian Optimization or molecular design.

Conclusion

The shift in this field is from generative modeling (making realistic samples) to representation engineering (extracting useful features). As Ruyi An notes in Improving Unsupervised Hierarchical Representation With Reinforcement Learning, models with lower likelihood can actually yield superior representations for downstream tasks. By treating the latent bottleneck as an information-theoretic channel to be managed rather than just a side-effect of reconstruction, these researchers are building models that truly "understand" the structure of the data they process.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_information_theoretic_constraints_for_us_a4c98fdb,
  title = {Information Theoretic Constraints for Useful Latent Representations},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/information-theoretic-constraints-for-useful-latent-representations/rdr_a4c98fdb560d4eaf81f642192918f8cd},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_information_theoretic_constraints_for_us_a4c98fdb_v1,
  title = {Information Theoretic Constraints for Useful Latent Representations},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/information-theoretic-constraints-for-useful-latent-representations/rdr_a4c98fdb560d4eaf81f642192918f8cd/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}