Standard information theory, pioneered by Claude Shannon, tells us how much information is shared between two variables. However, it remains silent on how that information is distributed. If two sensors provide information about a target, do they provide the same information (redundancy), different pieces of information (uniqueness), or information that only emerges when the two are combined (synergy)?
Partial Information Decomposition (PID) is a theoretical framework designed to "crack open" mutual information and reveal these hidden components. By partitioning information into distinct "atoms," researchers like William A. Phillips, Joseph T. Lizier, and Viola Priesemann argue we can finally distinguish between simple information relay and complex neural processing in both biological and artificial systems Partial information decomposition as a unified approach.
The Underdetermined Puzzle of Information Atoms
The core challenge of PID is that it is mathematically "underdetermined." As Aaron J. Gutknecht, Abdullah Makkeh, and Michael Wibral explain in From Babel to Boole, we know the total mutual information (the "whole"), but we have more unknown "atoms" (redundancy, synergy, and unique components) than we have independent equations to solve for them.
To resolve this, researchers have proposed various mathematical axioms to define a "redundancy" measure, but these often conflict. In Novel Inconsistency Results for Partial Information Decomposition, Abdullah Makkeh and colleagues prove that several intuitively desirable properties for these atoms are mathematically incompatible. This forces a trade-off: researchers must choose which logical properties to prioritize based on the system they are studying.
From Static Information to Dynamic Flow
While basic PID looks at static snapshots of data, Integrated Information Decomposition ($\Phi$ID) extends these concepts to time-series and dynamical systems. In Beyond integrated information, Pedro A. M. Mediano and Fernando E. Rosas introduce a taxonomy of 16 distinct atoms of information flow.
This framework allows scientists to see the "gears" of a system turning. For example, it can distinguish between:
- Information Storage: Information that a variable maintains over time.
- Information Transfer: Information moving from one variable to another.
- Synergistic Integration: Information that only exists in the collective dynamics of the whole system.
This has profound implications for neuroscience. Using $\Phi$ID, Alberto Liardi and Robin L. Carhart-Harris have shown that psychedelic states consistently shift brain dynamics toward higher synergy, suggesting a more "integrated" but less "redundant" mode of processing Null models for comparing information decomposition across complex systems.
Grounding Information in Logic and Geometry
To move away from arbitrary definitions, recent work has attempted to ground PID in formal logic and mereology (the study of parts and wholes). Aaron J. Gutknecht and Michael Wibral argue in Bits and pieces that information atoms correspond to a hierarchy of logical constraints. By viewing information through the lens of Boolean logic, researchers can use tools like Fourier analysis to interpret abstract concepts like "synergy" as specific types of sensitivity in a system's response Partial Information Decomposition of Boolean Functions.
Furthermore, Fernando E. Rosas and colleagues have introduced entropic conjugation, a principle that unifies the "zoo" of high-order information measures Characterising high-order interdependence via entropic conjugation. This mathematical symmetry helps identify which metrics, like O-information, are most efficient at balancing the tension between synergy and redundancy.
Scaling to Deep Learning and Neural Codes
As systems grow in complexity, traditional PID becomes computationally impossible due to the combinatorial explosion of atoms. To solve this, Aaron J. Gutknecht and Pedro A. M. Mediano proposed Shannon invariants Shannon invariants: A scalable approach to information decomposition. These global metrics capture the average degree of redundancy or vulnerability in a system, allowing for the analysis of high-order interactions in deep neural networks.
In biological systems, these tools are being used to decode the "language" of the brain. Giuseppe Pica and Christopher D. Harvey developed Intersection Information ($I_{II}$) to quantify how much sensory information is actually used to drive behavior, finding that precise spike timing is a key "behavioral code" Quantifying how much sensory information in a neural code is relevant for behavior. Similarly, Jan Bím and Andrea Brovelli introduced Feature-specific Information Transfer (FIT) to track the flow of specific sensory features across neural networks A Non-negative Measure Of Feature-Related Information Transfer.
By moving beyond "how much" information is present to "what kind" of information is being processed, Partial Information Decomposition provides a rigorous mathematical foundation for understanding the architecture of complexity.