Abstract
This paper provides a streamlined mathematical bridge between information theory and Bayesian inference by revisiting the optimality of Jeffrey’s update rule. While previous proofs of the rule's ability to minimize Kullback-Leibler (KL) divergence relied on complex linear algebra, this work offers a concise proof by reframing the update as a single iteration of the Expectation-Maximization (EM) algorithm. By leveraging the duality between likelihood maximization and divergence minimization, the authors provide a more intuitive and elegant foundation for understanding belief updates in probabilistic models.
Introduction
In Bayesian statistics, the process of updating internal beliefs in response to empirical observations is fundamental. We typically operate with an internal model, parameterized by $\theta$, which generates predictions about the world through a conditional probability channel $C(y|x)$. When new data arrives in the form of an empirical distribution $\tau$, the model must be updated to align more closely with this reality.
Jeffrey’s update rule is a classic method for this task. While it has long been understood that this rule is "optimal"—specifically that it minimizes the Kullback-Leibler (KL) divergence between the model's predictions and the observed data—the mathematical proofs for this optimality have historically been dense. Notably, researcher Bart Jacobs previously established this connection using heavy linear algebraic machinery. This paper seeks to simplify that proof, making the underlying logic accessible to a broader research community by utilizing the framework of optimization.
The EM Framework for Jeffrey’s Rule
The core innovation of Pinzón and Palamidessi lies in the recognition that Jeffrey’s update rule is not merely an isolated heuristic, but is mathematically equivalent to a single iteration of the Expectation-Maximization (EM) algorithm. This realization allows the authors to replace complex matrix manipulations with standard optimization arguments.
Mathematical Methodology
The proof of the theorem follows a structured four-step process:
- Likelihood-Divergence Duality: The authors first establish that increasing the log-likelihood of the model is mathematically equivalent to decreasing the KL divergence between the empirical observations $\tau$ and the model’s predictions $\vec{C}(\theta)$.
- Likelihood Decomposition: Following standard EM theory, the log-likelihood is decomposed into two distinct components: the $Q$ function (the expected complete-data log-likelihood) and the $H$ function (related to entropy).
- Application of Gibbs' Inequality: By applying Gibbs' inequality, the authors demonstrate that the $H$ component is naturally non-negative, ensuring that it does not detract from the optimization process.
- Optimization via Lagrange Multipliers: Finally, the method of Lagrange multipliers is used to prove that Jeffrey’s update rule is the unique mathematical solution that maximizes the $Q$ component.
By demonstrating that both the $Q$ and $H$ components of the likelihood improve (or at least remain constant) during the update, the authors provide a rigorous proof that the overall KL divergence must decrease.
Key Insights and Results
The primary result of this work is a simplified proof of Theorem 1, which states that Jeffrey’s update rule reduces (or maintains) the relative entropy $D_{KL}(\tau || \vec{C}(\theta))$. Quantitatively, the authors show that the change in log-likelihood ($\Delta L$) for the update is always $\ge 0$. Because $\Delta L$ is inversely related to the change in KL divergence, this confirms that the "error" in the model's predictions consistently shrinks when the rule is applied.
Beyond the proof itself, the paper offers several significant insights:
- The EM Connection: By linking Jeffrey’s rule to the EM algorithm, the authors connect a classic Bayesian tool to a vast body of existing work in machine learning and statistics.
- Robustness to Sparsity: The authors demonstrate that the rule remains valid even in "sparse" scenarios where certain input-output combinations have zero probability, provided the initial model does not assign zero probability to events that actually occurred (i.e., the model is not "pathological").
Discussion and Limitations
While the paper provides a more elegant and intuitive proof, it focuses on a specific context involving a fixed channel $C$. In many modern machine learning applications, the channel itself may be non-stationary or subject to learning. Furthermore, while the EM algorithm is a well-known framework, this paper does not extend the proof to more complex hierarchical Bayesian models or generalized update rules.
Conclusion
Pinzón and Palamidessi succeed in providing a "more simple and mathematically more elegant" proof of a foundational concept in Bayesian inference. By reframing Jeffrey’s update rule as an optimization step within the EM framework, they offer a clearer foundation for researchers working at the intersection of information theory and probabilistic learning.
Summary updated