In the intersection of information theory and additive combinatorics, a fundamental question has persisted: when do the entropy inequalities established for discrete variables apply to continuous ones? This paper, authored by Ashok Vardhan Makkuva and Yihong Wu, provides a definitive answer, establishing a unified framework for translating results across these domains.
The Challenge of Continuity
Additive combinatorics often explores how the sizes of sets behave under addition or subtraction. Information theory mirrors this by replacing set cardinality with Shannon entropy ($H(X)$) for discrete random variables. While many inequalities translate seamlessly, others fail. For instance, submodularity—a cornerstone of discrete entropy—does not hold for differential entropy ($h(X)$) in the continuous case.
Historically, researchers have had to prove continuous versions of discrete inequalities on a case-by-case basis, often using complex, specialized techniques. The goal of this work was to find a general principle that dictates which discrete entropy inequalities carry over to the continuous realm.
The Quantization Approach
The authors employ a quantization method to bridge the gap. By "chopping" a continuous random variable into increasingly fine discrete pieces, one can approximate its differential entropy using Shannon entropy. This relies on a classic result by Alfréd Rényi: $$H(\lfloor mX \rfloor) \approx d \log m + h(X)$$ where $m$ is the quantization factor.
The technical breakthrough in this paper lies in proving that the entropy of the sum of these quantized variables is asymptotically equal to the entropy of the quantized sum. By demonstrating that the error between these values vanishes as the quantization becomes infinitely fine ($m \to \infty$), the authors successfully passed discrete inequalities through the limit to derive their continuous counterparts.
The "Balanced" Requirement
The most significant insight of the paper is the identification of the "balanced" requirement. For a differential entropy inequality to be valid, it must be balanced—meaning the sum of its coefficients must be zero.
This necessity arises from the different ways Shannon and differential entropy respond to scaling. Shannon entropy is scale-invariant, whereas differential entropy is scale-dependent ($h(aX) = h(X) + d \log |a|$). If an inequality were not balanced, scaling the variables would lead to a mathematical contradiction. The authors prove that this condition is not just necessary, but also sufficient for a discrete inequality to hold in the continuous case.
Key Results and Implications
The paper's primary contribution is Theorem 1: any balanced affine inequality that holds for Shannon entropy across all independent discrete random variables also holds for differential entropy across all independent continuous random variables.
This result offers several key benefits:
- A Unified Translation Manual: It demystifies the parallel between discrete and continuous inequalities, showing that separate proof techniques are often unnecessary.
- Simplification of Prior Research: Several complex differential entropy inequalities discovered in recent years are revealed to be direct consequences of their discrete counterparts.
- Group Theory Generalization: The findings are extended beyond Euclidean space ($\mathbb{R}^d$) to connected abelian Lie groups, indicating that these relationships are rooted in fundamental algebraic and topological structures.
Limitations and Future Directions
While the paper provides a complete mapping from discrete to continuous for balanced inequalities, the reverse is not true. Certain inequalities, such as the Entropy Power Inequality, hold for differential entropy but have no direct Shannon entropy analog. This suggests that the continuous domain is more constrained than the discrete one.
Future research may explore whether these "balanced" principles apply to non-abelian groups or to more exotic entropy measures, such as Rényi entropy of orders other than 1. For now, this work stands as a definitive resolution to a central question in information-theoretic additive combinatorics.
Summary updated