Dissipation of Information in Channels With Input Constraints

Yury Polyanskiy, Yihong Wu

Abstract

This paper addresses the long-standing problem of quantifying information dissipation in long chains of power-constrained Gaussian channels. While information in discrete systems decays exponentially, we demonstrate that in Gaussian systems with average power constraints, information dissipates at a much slower logarithmic rate. We introduce the "Dobrushin curve" as a non-linear tool to characterize this process and show that non-linear relay schemes are essential for preserving information over long distances.

Introduction

In information theory, the Data Processing Inequality (DPI) establishes that information can only be lost as it passes through a sequence of processing steps, known as a Markov chain. For discrete channels, such as those involving bit-flips, this loss is typically aggressive, with the initial message's influence decaying exponentially toward zero.

However, for continuous channels with average power constraints—most notably the Additive White Gaussian Noise (AWGN) channel—standard tools like the Dobrushin contraction coefficient often fail. These coefficients frequently equal one, suggesting no dissipation occurs, which contradicts physical intuition. This paper provides a rigorous framework to quantify the "leakage" of information in systems where inputs are constrained by average energy rather than discrete symbol sets.

A sequence of relays $f_i$ and additive noise $Z_i$ forming a Markov chain $W \to Y_1 \to \dots \to Y_n$.

Methods: The Dobrushin Curve

To overcome the limitations of single-number contraction coefficients, we propose the Dobrushin curve ($F_{TV}$). This non-linear function maps the input total variation (a measure of statistical distance) to the maximum possible output total variation, subject to cost constraints such as power.

Key methodological components include:

  • Coupling Arguments: We utilize coupling techniques to construct joint probability distributions for different random variables, allowing for direct comparison in additive-noise channels.
  • Integral Representations: The findings are extended from Total Variation (TV) to more complex measures like Kullback-Leibler (KL) and Rényi divergences using mathematical identities that represent these divergences as integrals of the TV-based Dobrushin curve.
  • Non-linear Relay Schemes: To demonstrate the sharpness of our results, we design binary-messaging schemes using highly skewed quantization to maintain information over many hops.

Key Insights and Results

The most significant discovery is the fundamental difference in dissipation rates between discrete and power-constrained continuous channels.

Logarithmic vs. Exponential Decay

In a chain of $n$ relays, we prove that mutual information decays as $\Theta(\frac{\log \log n}{\log n})$. This logarithmic rate is remarkably slow compared to the $e^{-n}$ decay observed in discrete systems. This suggests that information is far more resilient in energy-constrained continuous systems than previously thought.

The Necessity of Non-linearity

While linear controllers are favored for their simplicity, they are mathematically suboptimal for long chains, leading to exponential information loss. Our work shows that non-linear "quantize-and-forward" schemes are necessary to achieve the optimal logarithmic dissipation rate.

Absence of Phase Transitions in Trees

Unlike discrete networks, which often exhibit a threshold where information vanishes completely if noise is too high, Gaussian channels with average power constraints show no such phase transition. Some information always persists at the leaves of a tree network, regardless of the Signal-to-Noise Ratio (SNR).

Summary of Scaling Laws

  • Total Variation: Scales as $O(1/\log n)$.
  • Mutual Information: Scales as $O(\frac{\log \log n}{\log n})$.
  • Maximal Squared Correlation: Scales as $O(\sqrt{\frac{\log \log n}{\log n}})$.

Limitations and Future Work

While this framework provides a robust foundation, several areas remain for exploration. The upper bounds for mutual information currently deteriorate as the signal dimension $d$ increases, suggesting that high-dimensional signals might experience faster dissipation. Additionally, while the framework is general, many concrete results are optimized for Gaussian noise; extending the Dobrushin curve analysis to heavy-tailed or other non-Gaussian distributions is a natural next step. Finally, the contraction of KL divergence under specific moment constraints requires further investigation to ensure universal applicability across all divergence types.

Summary updated

Abstract

One of the basic tenets in information theory, the data processing inequality states that output divergence does not exceed the input divergence for any channel. For channels without input constraints, various estimates on the amount of such contraction are known, Dobrushin's coefficient for the total variation being perhaps the most well-known. This work investigates channels with average input cost constraint. It is found that while the contraction coefficient typically equals one (no contraction), the information nevertheless dissipates. A certain non-linear function, the \emph{Dobrushin curve} of the channel, is proposed to quantify the amount of dissipation. Tools for evaluating the Dobrushin curve of additive-noise channels are developed based on coupling arguments. Some basic applications in stochastic control, uniqueness of Gibbs measures and fundamental limits of noisy circuits are discussed. As an application, it shown that in the chain of $n$ power-constrained relays and Gaussian channels the end-to-end mutual information and maximal squared correlation decay as $Θ(\frac{\log\log n}{\log n})$, which is in stark contrast with the exponential decay in chains of discrete channels. Similarly, the behavior of noisy circuits (composed of gates with bounded fan-in) and broadcasting of information on trees (of bounded degree) does not experience threshold behavior in the signal-to-noise ratio (SNR). Namely, unlike the case of discrete channels, the probability of bit error stays bounded away from $1\over 2$ regardless of the SNR.
BibTeX
@article{polyanskiy2014dissipation,
  author = {Yury Polyanskiy and Yihong Wu},
  title = {{Dissipation of Information in Channels With Input Constraints}},
  journal = {IEEE Trans. Information Theory},
  year = {2014},
  url = {https://openreview.net/forum?id=YBuhzqvXiVc},
}
Figure