Abstract
This paper addresses the long-standing problem of quantifying information dissipation in long chains of power-constrained Gaussian channels. While information in discrete systems decays exponentially, we demonstrate that in Gaussian systems with average power constraints, information dissipates at a much slower logarithmic rate. We introduce the "Dobrushin curve" as a non-linear tool to characterize this process and show that non-linear relay schemes are essential for preserving information over long distances.
Introduction
In information theory, the Data Processing Inequality (DPI) establishes that information can only be lost as it passes through a sequence of processing steps, known as a Markov chain. For discrete channels, such as those involving bit-flips, this loss is typically aggressive, with the initial message's influence decaying exponentially toward zero.
However, for continuous channels with average power constraints—most notably the Additive White Gaussian Noise (AWGN) channel—standard tools like the Dobrushin contraction coefficient often fail. These coefficients frequently equal one, suggesting no dissipation occurs, which contradicts physical intuition. This paper provides a rigorous framework to quantify the "leakage" of information in systems where inputs are constrained by average energy rather than discrete symbol sets.
Methods: The Dobrushin Curve
To overcome the limitations of single-number contraction coefficients, we propose the Dobrushin curve ($F_{TV}$). This non-linear function maps the input total variation (a measure of statistical distance) to the maximum possible output total variation, subject to cost constraints such as power.
Key methodological components include:
- Coupling Arguments: We utilize coupling techniques to construct joint probability distributions for different random variables, allowing for direct comparison in additive-noise channels.
- Integral Representations: The findings are extended from Total Variation (TV) to more complex measures like Kullback-Leibler (KL) and Rényi divergences using mathematical identities that represent these divergences as integrals of the TV-based Dobrushin curve.
- Non-linear Relay Schemes: To demonstrate the sharpness of our results, we design binary-messaging schemes using highly skewed quantization to maintain information over many hops.
Key Insights and Results
The most significant discovery is the fundamental difference in dissipation rates between discrete and power-constrained continuous channels.
Logarithmic vs. Exponential Decay
In a chain of $n$ relays, we prove that mutual information decays as $\Theta(\frac{\log \log n}{\log n})$. This logarithmic rate is remarkably slow compared to the $e^{-n}$ decay observed in discrete systems. This suggests that information is far more resilient in energy-constrained continuous systems than previously thought.
The Necessity of Non-linearity
While linear controllers are favored for their simplicity, they are mathematically suboptimal for long chains, leading to exponential information loss. Our work shows that non-linear "quantize-and-forward" schemes are necessary to achieve the optimal logarithmic dissipation rate.
Absence of Phase Transitions in Trees
Unlike discrete networks, which often exhibit a threshold where information vanishes completely if noise is too high, Gaussian channels with average power constraints show no such phase transition. Some information always persists at the leaves of a tree network, regardless of the Signal-to-Noise Ratio (SNR).
Summary of Scaling Laws
- Total Variation: Scales as $O(1/\log n)$.
- Mutual Information: Scales as $O(\frac{\log \log n}{\log n})$.
- Maximal Squared Correlation: Scales as $O(\sqrt{\frac{\log \log n}{\log n}})$.
Limitations and Future Work
While this framework provides a robust foundation, several areas remain for exploration. The upper bounds for mutual information currently deteriorate as the signal dimension $d$ increases, suggesting that high-dimensional signals might experience faster dissipation. Additionally, while the framework is general, many concrete results are optimized for Gaussian noise; extending the Dobrushin curve analysis to heavy-tailed or other non-Gaussian distributions is a natural next step. Finally, the contraction of KL divergence under specific moment constraints requires further investigation to ensure universal applicability across all divergence types.
Summary updated