Abstract
Many of the classical and recent relations between information and estimation in the presence of Gaussian noise can be viewed as identities between expectations of random quantities. These include the I-MMSE relationship of Guo et al.; the relative entropy and mismatched estimation relationship of Verdú; the relationship between causal estimation and mutual information of Duncan, and its extension to the presence of feedback by Kadota et al.; and the relationship between causal and non-casual estimation of Guo et al. This paper dispenses with the expectations and explores the nature of the pointwise relations between the respective random quantities. We find that these pointwise relations are as succinctly stated as the original expectation identities and provide considerable insight into the underlying stochastic structures. Specifically, we characterize the difference between information density and estimation error as a stochastic integral, revealing that the variance of this random variable is itself given by the causal MMSE.
Introduction
Information theory and estimation theory are deeply intertwined, particularly in the context of additive white Gaussian noise (AWGN) channels. Elegant formulas like Duncan’s theorem state that mutual information equals half the causal minimum mean squared error (MMSE), while the I-MMSE theorem links the derivative of mutual information to the non-causal MMSE.
However, these classical results are expectation identities. They describe what happens on average across all possible realizations of signals and noise. This research addresses a fundamental question: what happens before the average is taken? By examining the "pointwise" relationships between information density and squared estimation error for specific realizations, we can better understand how closely these quantities track each other in real-time.
Methodology: Stochastic Calculus and Martingales
The study utilizes advanced continuous-time stochastic calculus to model the AWGN channel as a Brownian motion process. The authors employ several key mathematical frameworks:
- Girsanov Theory & Itô Calculus: By expressing information density (the log Radon-Nikodym derivative) and estimation error within this framework, the difference between them—termed the "tracking error"—is represented as a stochastic integral with respect to the driving Brownian motion.
- Noise Coupling: To analyze how these variables evolve across different Signal-to-Noise Ratios (SNRs), the authors "couple" the noise across SNRs using a standard Brownian motion or Brownian sheet. This allows for the comparison of different SNR levels on the same probability space.
- Martingale Analysis: By proving that the tracking error is an Itô integral, the authors establish that it is a martingale (a process with a constant expectation of zero). This structural property inherently explains why taking the expectation recovers the classical theorems.
Key Insights and Results
The Tracking Error as a Martingale
The research proves that the tracking error $Z$ between information density and half the integrated squared error over SNR is explicitly characterized as: $$Z = \int (X - E[X|Y_\gamma]) dW_\gamma$$ This identity shows that the difference is not merely random noise but a structured stochastic integral.
The Variance Identity
A major discovery of this work is that the variance of this tracking error has a beautiful, distribution-independent form. In the context of Duncan's theorem, the variance of the tracking error is exactly equal to the causal MMSE (which is also twice the mutual information): $$Var(Z) = \int mmse(\gamma) d\gamma = 2I(X; Y_{snr})$$ This implies that the fluctuations of the error are perfectly described by the very quantities being estimated, a result that holds regardless of the distribution of the channel input.
Generalization to Mismatch and Feedback
The pointwise approach generalizes beyond ideal scenarios to include:
- Mismatched Estimation: Where the receiver assumes an incorrect statistical distribution for the signal.
- Feedback Channels: Where the input depends on previous output realizations. In both cases, the authors find that the tracking error maintains its martingale structure and yields similar variance identities.
Causal vs. Anti-causal Estimation
The paper provides a pointwise characterization showing that the difference between causal and anti-causal estimation errors is the difference between two zero-mean stochastic integrals. This explains structurally why these two seemingly different estimation approaches yield equal values in expectation.
Limitations and Future Directions
The current findings are strictly focused on Gaussian noise channels. While the scalar case and specific Gaussian inputs are well-explored, the continuous-time coupling variations remain an area for further study. The authors suggest that future research should investigate higher-order moments of the tracking error and explore whether similar pointwise relationships exist in other channel models, such as the Poisson channel, where recent links between estimation and information have been identified.
Conclusion
By shifting the focus from expectations to pointwise realizations, this research uncovers a deeper layer of the relationship between information and estimation. The discovery that tracking error is a martingale with a variance tied directly to the MMSE provides a new mathematical lens through which to view the stability and behavior of information-theoretic quantities in Gaussian noise.
Summary updated