Information Theoretic lower bounds on negative log likelihood

Luis Alfonso Lastras-Montaño

When developing latent variable models like Variational Autoencoders (VAEs), researchers frequently iterate on the prior (the distribution of hidden variables) and the likelihood function (how variables generate data). A persistent challenge is diagnosing a poor Negative Log-Likelihood (NLL) score: is it caused by an inadequate prior, a flawed likelihood function, or has the model simply reached its theoretical limit?

This paper bridges latent variable modeling and rate-distortion theory to provide a mathematical "yardstick" for NLL. By deriving information-theoretic lower bounds, the author provides a formal method to quantify exactly how much a model could improve if its prior were optimized in an unrestricted way, helping researchers pinpoint where to invest their engineering efforts.

The Theoretical Framework

The core contribution of the paper is a formal proof demonstrating that optimizing a prior in a latent variable model is mathematically equivalent to computing rate-distortion functions in classical information theory.

  • Theorem 1 (Equivalence): The author shows that the optimization problem for finding the best prior is an instance of the variational optimization used in rate-distortion theory.
  • Theorem 2 (The Lower Bound): Leveraging this equivalence, the paper derives a "sharp" lower bound for the average NLL. This bound relies on a novel quantity, $c(z)$, which measures the ratio between the likelihood and the model's marginal probability. The bound is considered "sharp" because if the mathematically optimal prior is applied, the lower bound perfectly matches the actual NLL.

Practical Application: "Glossy Statistics"

Because finding the absolute best prior in continuous spaces is often computationally intractable, the paper introduces "glossy statistics"—specifically, the maximum and standard deviation of $\log c(z)$.

These statistics serve as practical indicators of a model's distance from its theoretical optimum. The quantity $c(z)$ acts as a multiplicative factor; if $c(z)$ is not constant (i.e., its standard deviation is high), it proves that the model is sub-optimal and there is "slack" in the NLL that can be recovered.

Experimental Validation and Insights

The author validates these bounds across a synthetic dataset (with a known true latent model to track convergence) and standard benchmarks including Static MNIST, OMNIGLOT, Caltech 101 Silhouettes, Frey Faces, Histopathology, and CIFAR-10. Several key insights emerged from these experiments:

  • Prior vs. Likelihood Symmetry: A surprising finding is that if modifying the prior improves the log-likelihood, there is almost always a corresponding modification to the likelihood function that achieves the exact same improvement. This demonstrates that rate-distortion theory is relevant to both sides of the modeling equation.
  • Detecting Posterior Collapse: In Histopathology experiments using a PixelHVAE model, the glossy statistics dropped to nearly zero. This provided rigorous theoretical confirmation of posterior collapse, a known issue where powerful decoders learn to completely ignore latent variables.
  • Guiding Model Design: On MNIST, standard VAEs exhibited relatively large glossy statistics. This correctly predicted that adopting a more complex prior (like VampPrior) or a superior architecture (like HVAE) would yield measurable NLL improvements. Furthermore, on the synthetic dataset, the gap between the upper and lower bounds visibly narrowed during training, illustrating the model successfully closing in on the true NLL.

Limitations and Future Directions

The approach is not without challenges. In continuous latent spaces, calculating the exact maximum of $c(z)$ is computationally difficult, necessitating sampling-based approximations that can introduce variance depending on the sampling method. Additionally, while the theory identifies when improvement is possible via a change of variables, it does not necessarily rule out other, more complex types of likelihood improvements.

Ultimately, by translating the "crown jewels" of information theory into actionable metrics for machine learning, this work provides a rigorous diagnostic tool for generative modeling. The author suggests that future work could apply other foundational concepts, such as Shannon’s source coding theorem and multi-terminal information theory, to further demystify complex machine learning bottlenecks.

Summary updated

Abstract

In this article we use rate-distortion theory, a branch of information theory devoted to the problem of lossy compression, to shed light on an important problem in latent variable modeling of data: is there room to improve the model? One way to address this question is to find an upper bound on the probability (equivalently a lower bound on the negative log likelihood) that the model can assign to some data as one varies the prior and/or the likelihood function in a latent variable model. The core of our contribution is to formally show that the problem of optimizing priors in latent variable models is exactly an instance of the variational optimization problem that information theorists solve when computing rate-distortion functions, and then to use this to derive a lower bound on negative log likelihood. Moreover, we will show that if changing the prior can improve the log likelihood, then there is a way to change the likelihood function instead and attain the same log likelihood, and thus rate-distortion theory is of relevance to both optimizing priors as well as optimizing likelihood functions. We will experimentally argue for the usefulness of quantities derived from rate-distortion theory in latent variable modeling by applying them to a problem in image modeling.
BibTeX
@inproceedings{lastrasmontano2019information,
  author = {Luis Alfonso Lastras-Montaño},
  title = {{Information Theoretic lower bounds on negative log likelihood}},
  booktitle = {International Conference on Learning Representations},
  year = {2019},
}
Figure