Abstract
The Fisher Information Inequality (FII) is a fundamental result in statistics and information theory, yet its traditional proofs often lack intuitive physical grounding. This paper presents two alternative proofs that frame the FII as a direct consequence of the Data Processing Inequality (DPI). By shifting the perspective from abstract calculus to communication and estimation theory, the authors provide an operational interpretation of Fisher information as a measure of information content that cannot be increased through processing.
Introduction
The Fisher Information Inequality (FII) serves as a cornerstone for several major results in information theory, including the Entropy Power Inequality (EPI) and the Central Limit Theorem (CLT). In its canonical form, it states that for independent random variables $N_1$ and $N_2$:
$$\frac{1}{J(N_1 + N_2)} \geq \frac{1}{J(N_1)} + \frac{1}{J(N_2)}$$
where $J(\cdot)$ denotes the Fisher information. While mathematically sound, classical proofs—such as the Stam-Blachman proof—rely heavily on the Cauchy-Schwartz inequality. These approaches are often criticized for being mathematically "dry" and failing to explain why the inequality holds in a physical or information-theoretic context. This paper addresses this gap by demonstrating that the FII is a natural manifestation of the Data Processing Principle: the concept that signal processing cannot increase the information content of a signal.
The Data Processing Framework
The core of the authors' argument lies in the construction of a specific Markov chain:
$$W \to (X_1, X_2) \to X_1 + X_2$$
In this model, $W$ represents a signal, $(X_1, X_2)$ represents the data collected, and $X_1 + X_2$ represents a processed (summed) version of that data. According to the Data Processing Inequality, the information about $W$ contained in the sum $X_1 + X_2$ must be less than or equal to the information contained in the individual components $(X_1, X_2)$. The authors show that the FII is simply a mathematical expression of this loss of information during the summation process.
Method 1: The Communications Proof
The first proof utilizes the Data Processing Inequality for Mutual Information, $I(W; Y) \leq I(W; X)$. The authors establish a communication model where a Gaussian signal $W$ is transmitted through two parallel channels subject to independent noise.
To link mutual information to Fisher information, the authors employ the De-Bruijn identity. This identity relates the derivative of the mutual information between a signal and a noise-corrupted version of that signal to the Fisher information of the noise. By examining the signal-to-noise ratio as it approaches zero, the authors demonstrate that the FII emerges naturally from the limits of channel capacity. This approach operationalizes Fisher information as a limit on the rate of information transfer.
Method 2: The Bayesian Estimation Proof
The second proof shifts the focus to estimation theory, specifically the DPI for Conditional Variance. This principle posits that the minimum mean square error (MMSE) of an estimate cannot decrease if the estimator only has access to a processed version of the observed data.
A key contribution of this method is Lemma 1, which provides a clean relationship between the Fisher information of a noisy variable and the conditional variance (or MMSE) of estimating the noise:
$$J(N+W) = \frac{1}{\sigma^4} \{ \sigma^2 - \text{VAR}[N|N+W] \}$$
By applying this lemma to the sum of two random variables, the authors show that the FII is essentially a statement about the fundamental limits of Bayesian estimation. If you "mix" two noisy signals by adding them, your ability to estimate the underlying noise components is strictly worse than if you had access to the signals separately.
Key Insights and Implications
Beyond providing more intuitive proofs, this framework offers several theoretical advantages:
- Operational Meaning: Fisher information is redefined as a measure of "processing loss," moving it out of the realm of pure statistics.
- Generalization: Unlike classical methods, the data processing argument naturally extends the FII to cases involving dependent random variables, a result that is significantly harder to derive using traditional calculus.
- Unified Theory: The paper demonstrates that the FII is not an isolated statistical property but belongs to a broader family of information-theoretic inequalities governed by the Data Processing Principle.
Limitations and Conclusion
While the proofs offer significant clarity, they do introduce certain complexities. In classical proofs, the condition for equality (which occurs if and only if the variables are Gaussian) is explicit. In these new proofs, the necessity of Gaussianity is less apparent because the results rely on limiting arguments (e.g., as a parameter $t \to 0$). Additionally, while the communications proof can be generalized to non-Gaussian signals matching specific moments, the estimation proof remains more strictly tied to Gaussian perturbations.
In conclusion, by reframing the Fisher Information Inequality through the lens of data processing, the authors bridge the gap between pure statistics and information theory, providing a more robust and intuitive understanding of a fundamental limit in the field.
Summary updated