Pointwise Relations between Information and Estimation

Kartik Venkat, Tsachy Weissman

Abstract

Many of the classical and recent relations between information and estimation in the presence of Gaussian noise can be viewed as identities between expectations of random quantities. These include the I-MMSE relationship of Guo et al.; the relative entropy and mismatched estimation relationship of Verdú; the relationship between causal estimation and mutual information of Duncan, and its extension to the presence of feedback by Kadota et al.; and the relationship between causal and non-casual estimation of Guo et al. This paper dispenses with the expectations and explores the nature of the pointwise relations between the respective random quantities. We find that these pointwise relations are as succinctly stated as the original expectation identities and provide considerable insight into the underlying stochastic structures. Specifically, we characterize the difference between information density and estimation error as a stochastic integral, revealing that the variance of this random variable is itself given by the causal MMSE.

Introduction

Information theory and estimation theory are deeply intertwined, particularly in the context of additive white Gaussian noise (AWGN) channels. Elegant formulas like Duncan’s theorem state that mutual information equals half the causal minimum mean squared error (MMSE), while the I-MMSE theorem links the derivative of mutual information to the non-causal MMSE.

However, these classical results are expectation identities. They describe what happens on average across all possible realizations of signals and noise. This research addresses a fundamental question: what happens before the average is taken? By examining the "pointwise" relationships between information density and squared estimation error for specific realizations, we can better understand how closely these quantities track each other in real-time.

Methodology: Stochastic Calculus and Martingales

The study utilizes advanced continuous-time stochastic calculus to model the AWGN channel as a Brownian motion process. The authors employ several key mathematical frameworks:

  • Girsanov Theory & Itô Calculus: By expressing information density (the log Radon-Nikodym derivative) and estimation error within this framework, the difference between them—termed the "tracking error"—is represented as a stochastic integral with respect to the driving Brownian motion.
  • Noise Coupling: To analyze how these variables evolve across different Signal-to-Noise Ratios (SNRs), the authors "couple" the noise across SNRs using a standard Brownian motion or Brownian sheet. This allows for the comparison of different SNR levels on the same probability space.
  • Martingale Analysis: By proving that the tracking error is an Itô integral, the authors establish that it is a martingale (a process with a constant expectation of zero). This structural property inherently explains why taking the expectation recovers the classical theorems.

Key Insights and Results

The Tracking Error as a Martingale

The research proves that the tracking error $Z$ between information density and half the integrated squared error over SNR is explicitly characterized as: $$Z = \int (X - E[X|Y_\gamma]) dW_\gamma$$ This identity shows that the difference is not merely random noise but a structured stochastic integral.

The Variance Identity

A major discovery of this work is that the variance of this tracking error has a beautiful, distribution-independent form. In the context of Duncan's theorem, the variance of the tracking error is exactly equal to the causal MMSE (which is also twice the mutual information): $$Var(Z) = \int mmse(\gamma) d\gamma = 2I(X; Y_{snr})$$ This implies that the fluctuations of the error are perfectly described by the very quantities being estimated, a result that holds regardless of the distribution of the channel input.

Generalization to Mismatch and Feedback

The pointwise approach generalizes beyond ideal scenarios to include:

  • Mismatched Estimation: Where the receiver assumes an incorrect statistical distribution for the signal.
  • Feedback Channels: Where the input depends on previous output realizations. In both cases, the authors find that the tracking error maintains its martingale structure and yields similar variance identities.

Causal vs. Anti-causal Estimation

The paper provides a pointwise characterization showing that the difference between causal and anti-causal estimation errors is the difference between two zero-mean stochastic integrals. This explains structurally why these two seemingly different estimation approaches yield equal values in expectation.

Limitations and Future Directions

The current findings are strictly focused on Gaussian noise channels. While the scalar case and specific Gaussian inputs are well-explored, the continuous-time coupling variations remain an area for further study. The authors suggest that future research should investigate higher-order moments of the tracking error and explore whether similar pointwise relationships exist in other channel models, such as the Poisson channel, where recent links between estimation and information have been identified.

Conclusion

By shifting the focus from expectations to pointwise realizations, this research uncovers a deeper layer of the relationship between information and estimation. The discovery that tracking error is a martingale with a variance tied directly to the MMSE provides a new mathematical lens through which to view the stability and behavior of information-theoretic quantities in Gaussian noise.

Summary updated

Abstract

Many of the classical and recent relations between information and estimation in the presence of Gaussian noise can be viewed as identities between expectations of random quantities. These include the I-MMSE relationship of Guo et al.; the relative entropy and mismatched estimation relationship of Verd\'{u}; the relationship between causal estimation and mutual information of Duncan, and its extension to the presence of feedback by Kadota et al.; the relationship between causal and non-casual estimation of Guo et al., and its mismatched version of Weissman. We dispense with the expectations and explore the nature of the pointwise relations between the respective random quantities. The pointwise relations that we find are as succinctly stated as - and give considerable insight into - the original expectation identities. As an illustration of our results, consider Duncan's 1970 discovery that the mutual information is equal to the causal MMSE in the AWGN channel, which can equivalently be expressed saying that the difference between the input-output information density and half the causal estimation error is a zero mean random variable (regardless of the distribution of the channel input). We characterize this random variable explicitly, rather than merely its expectation. Classical estimation and information theoretic quantities emerge with new and surprising roles. For example, the variance of this random variable turns out to be given by the causal MMSE (which, in turn, is equal to the mutual information by Duncan's result).
BibTeX
@misc{venkat2011pointwise,
  author = {Kartik Venkat and Tsachy Weissman},
  title = {{Pointwise Relations between Information and Estimation}},
  year = {2011},
  eprint = {1110.6654},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1110.6654},
}
Figure