Comment on the Equality Condition for the I-MMSE Proof of Entropy Power Inequality

Alex Dytso, Ronit Bustin, H. Vincent Poor, Shlomo ShamaiarXiv 2017 · arXiv

Abstract

The Entropy Power Inequality (EPI) is a fundamental result in information theory, relating the differential entropy of independent random variables to the entropy of their sum. While the I-MMSE relationship provides an elegant framework for proving the EPI, the characterization of its equality condition within this framework has remained incomplete. This paper addresses this gap by deriving an exact expression for the EPI "deficit"—the difference between the two sides of the inequality. By leveraging the Cauchy functional equation, the authors rigorously demonstrate that equality holds if and only if the involved random variables are Gaussian with identical covariances.

Introduction

The Entropy Power Inequality (EPI), first introduced by Claude Shannon in 1948, is a cornerstone of information theory. It provides a lower bound on the entropy power of the sum of independent random variables, essentially stating that the "uncertainty" of a sum is at least the sum of the individual uncertainties. Mathematically, for independent random vectors $X_1$ and $X_2$, the EPI is often expressed as:

$$e^{2h(X_1+X_2)/n} \geq e^{2h(X_1)/n} + e^{2h(X_2)/n}$$

where $h(\cdot)$ denotes differential entropy. Over the decades, various proofs of the EPI have been developed using techniques ranging from integration by parts to Young’s inequality. In 2005, the introduction of the I-MMSE relationship—which connects Information Theory (mutual information) to Estimation Theory (Minimum Mean Square Error)—offered a new, elegant proof method. However, while the I-MMSE proof successfully established the inequality, it lacked a rigorous characterization of the equality condition. This paper provides the "missing piece of the puzzle," completing the I-MMSE-based proof of the EPI.

Methodology: The Deficit-Based Approach

The authors employ a "deficit-based" approach to the problem. Rather than simply proving that one side of the inequality is greater than or equal to the other, they establish an exact integral expression for the difference between the two sides of Lieb’s version of the EPI. This difference is referred to as the "deficit."

Key mathematical tools used in this derivation include:

  • The I-MMSE Relationship: Utilizing the fact that differential entropy can be expressed as an integral of the MMSE over a Gaussian noise channel.
  • Score Functions: Relating the gradient of the log-density to conditional expectations in Gaussian noise.
  • Properties of Conditional Expectation: Specifically the law of iterated expectations (towering property) to manipulate complex estimators.

By expressing the deficit as an expectation of a squared difference—a quantity that is inherently non-negative—the EPI follows naturally. More importantly, this explicit formula allows for a precise investigation into the specific conditions under which the deficit vanishes.

Characterizing Equality via Cauchy's Functional Equation

The core challenge in characterizing the equality condition lies in proving that the deficit is zero if and only if the random variables are Gaussian. The authors bridge a significant historical gap by applying the Cauchy functional equation ($f(x+y) = f(x) + f(y)$), a 19th-century mathematical tool, to this modern information-theoretic problem.

The proof demonstrates that the deficit is zero only if certain conditional expectations (estimators) are affine—meaning they are linear functions of the observations plus a constant. In the context of MMSE estimation, it is a well-known property that linear estimators are optimal if and only if the underlying distributions are Gaussian. By showing that the additivity of these estimators satisfies the Cauchy functional equation, the authors rigorously conclude that the only distributions satisfying the equality condition are Gaussian distributions with identical covariance matrices.

Key Results

The paper presents several significant theoretical contributions:

  • Theorem 1 (The Deficit Formula): Establishes the exact expression for the EPI deficit. Because this gap is expressed as an expectation of a squared difference, the inequality $h(\dots) \geq \dots$ is proven directly.
  • Theorem 2 (Equivalent Conditions): Provides six equivalent conditions for equality. These range from the deficit being zero to specific identities involving score functions and Fisher Information.
  • The Equality Condition: The paper rigorously concludes that for independent continuous random vectors $X_1$ and $X_2$, equality in the EPI holds if and only if $X_1$ and $X_2$ are Gaussian with identical covariances.
  • Fisher Information Inequality: The results also settle the equality condition for a related Fisher Information inequality, which is a key step in many EPI proofs.

Conclusion and Future Work

By closing the gap in the I-MMSE proof, this work ensures that this modern approach is as theoretically robust as classical proofs based on integration or functional analysis. The connection between 19th-century mathematics (Cauchy’s equation) and modern information-theoretic relationships (I-MMSE) provides a deep insight into the fundamental nature of Gaussianity.

While the proof assumes random vectors have covariance matrices with bounded entries and differentiable probability densities, these are standard assumptions in the field. For future work, the authors suggest exploring connections between their exact deficit characterization and the Wasserstein distance. This could lead to more "robust" versions of the EPI that quantify how close a distribution is to being Gaussian based on how close it comes to satisfying the equality condition, particularly for log-concave densities.

Summary updated

Abstract

The paper establishes the equality condition in the I-MMSE proof of the entropy power inequality (EPI). This is done by establishing an exact expression for the deficit between the two sides of the EPI. Interestingly, a necessary condition for the equality is established by making a connection to the famous Cauchy functional equation.
BibTeX
@misc{dytso2017comment,
  author = {Alex Dytso and Ronit Bustin and H. Vincent Poor and Shlomo Shamai},
  title = {{Comment on the Equality Condition for the I-MMSE Proof of Entropy Power Inequality}},
  year = {2017},
  eprint = {1703.07442},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1703.07442},
}
Figure