Notes on Kullback-Leibler Divergence and Likelihood

Jonathon ShlensarXiv 2014 · arXiv

Abstract

This paper provides a pedagogical bridge between information theory and statistics by demonstrating that Kullback-Leibler (KL) divergence is not merely an abstract "coding penalty," but a direct consequence of likelihood theory. By examining the asymptotic behavior of multinomial distributions, we show that KL divergence is the negative logarithm of the average likelihood of a model $q$ given an infinite stream of data generated by a true distribution $p$. This derivation provides a rigorous statistical justification for the use of KL divergence as a loss function and distance metric in machine learning and neuroscience.

Introduction

Kullback-Leibler (KL) divergence is a cornerstone of modern information theory, appearing in contexts ranging from Variational Autoencoder (VAE) loss functions to the analysis of neural population codes. Traditionally, it is introduced through the lens of coding theory: the number of extra bits required to encode a signal if one assumes a distribution $q$ when the true distribution is $p$. While mathematically elegant, this interpretation can feel abstract to scientists who primarily reason about data through the framework of likelihood.

In this note, we address this conceptual gap by providing an intuitive derivation of KL divergence. We tackle a fundamental question: If a researcher possesses a model $q$ and observes data generated by a true process $p$, how does the probability of that model being "correct" relate to the standard KL divergence formula?

Theoretical Framework

The derivation relies on a theoretical approach using discrete random variables and multinomial likelihoods. The process involves four primary mathematical steps:

  1. Multinomial Likelihood: We begin with the probability of observing a specific histogram of outcomes (e.g., dice rolls or binned neural firing patterns) given a candidate model $q$. This is expressed through the multinomial distribution.
  2. Average Likelihood ($\bar{L}$): To ensure the likelihood is invariant to the total number of measurements $n$, we utilize the geometric mean, defined as $\bar{L} = L^{1/n}$. This represents the "per-measurement" probability of the model.
  3. Stirling’s Approximation: To simplify the factorials inherent in multinomial coefficients as $n$ becomes very large, we apply Stirling’s approximation: $\ln n! \approx n \ln n - n$.
  4. Asymptotic Limit: By taking the limit as $n \to \infty$, the discrete counts in the observed histogram converge into the continuous probabilities of the true distribution $p$.

Key Insights

The Emergence of the KL Formula

The primary "Aha!" moment of this derivation is that the KL divergence formula, $\sum p_i \log \frac{p_i}{q_i}$, is not an arbitrary construction. It emerges naturally when calculating the average log-likelihood of a model. Specifically, the derivation proves the following relationship:

$$D_{KL}(p || q) = \lim_{n \to \infty} -\frac{1}{n} \log L(c|q)$$

In plain English, KL divergence is the negative log of the average multinomial likelihood. This provides a formal proof that minimizing KL divergence is mathematically equivalent to maximizing the average likelihood of a model given the true data distribution.

Entropy as Combinatorics

A significant side-effect of this derivation is the insight it provides into the nature of entropy. The formula for entropy ($-\sum p_i \log p_i$) arises directly from the normalization constant of the multinomial distribution. This suggests that entropy is fundamentally rooted in combinatorics—the number of ways different outcomes can be combined—rather than just being a measure of "disorder."

Likelihood as Proximity

The relationship defines a clear gradient of model fit. If a model $q$ is a perfect match for the data-generating process $p$, the average likelihood $\bar{L}$ approaches 1, and the KL divergence becomes 0. Conversely, as the model's fit degrades, the likelihood vanishes toward 0, and the KL divergence increases toward infinity.

Applications and Limitations

In the context of neuroscience, these tools are particularly powerful for measuring "conditional dependence." By comparing the KL divergence of models that assume independent firing versus those that account for correlations, researchers can quantify the information lost by ignoring population dynamics.

However, there are limitations to this specific derivation:

  • Discrete Focus: The use of multinomial likelihoods is inherently discrete-centric. While the intuition extends to continuous distributions, the combinatorial derivation presented here is specific to binned or discrete data.
  • Scope: This note focuses on the standard KL divergence and does not explore complex variants such as Rényi or Jensen-Shannon divergences, nor does it address non-i.i.d. (independent and identically distributed) settings.

Conclusion

By reframing KL divergence as a consequence of likelihood theory, we bridge the gap between information theory and frequentist statistics. This perspective justifies the ubiquity of KL divergence in optimization problems: it is the natural metric for researchers seeking to maximize the likelihood of their models in the face of empirical data.

Summary updated

Abstract

The Kullback-Leibler (KL) divergence is a fundamental equation of information theory that quantifies the proximity of two probability distributions. Although difficult to understand by examining the equation, an intuition and understanding of the KL divergence arises from its intimate relationship with likelihood theory. We discuss how KL divergence arises from likelihood theory in an attempt to provide some intuition and reserve a rigorous (but rather simple) derivation for the appendix. Finally, we comment on recent applications of KL divergence in the neural coding literature and highlight its natural application.
BibTeX
@misc{shlens2014notes,
  author = {Jonathon Shlens},
  title = {{Notes on Kullback-Leibler Divergence and Likelihood}},
  year = {2014},
  eprint = {1404.2000},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1404.2000},
}
Figure