Partial Information Decomposition

562 concepts259 papers2 proposals

Beyond the Limits of Mutual Information

Suppose you want to predict a patient's health outcome ($Y$) using two diagnostic tests: a blood panel ($X_1$) and a genetic screening ($X_2$). Classical information theory provides a powerful tool for this: Mutual Information, denoted as $I(X_1, X_2; Y)$. This metric tells us exactly how many "bits" of uncertainty about the patient's health are eliminated when we observe both tests together.

However, Shannon's mutual information treats the pair $(X_1, X_2)$ as a single joint variable. It cannot tell us how that information is distributed between the two sources. Do the blood panel and genetic test provide the exact same overlapping clues? Does the genetic test reveal something completely exclusive? Or do the two tests only make sense when cross-referenced against each other, revealing a risk factor that neither could show alone?

Standard information theory conflates these distinct dynamics into a single aggregate number. To peer inside this black box, researchers use Partial Information Decomposition (PID).

The Anatomy of Information Atoms

PID expands joint mutual information by partitioning it into distinct, non-negative "atoms" of information:

  1. Redundant Information ($R$): Overlapping information that can be extracted from either $X_1$ or $X_2$ independently.
  2. Unique Information ($U_1$): Information about $Y$ that is present only in $X_1$.
  3. Unique Information ($U_2$): Information about $Y$ that is present only in $X_2$.
  4. Synergistic Information ($S$): Emergent information that is completely invisible when looking at $X_1$ or $X_2$ individually, but appears when they are observed simultaneously.

Mathematically, the total joint mutual information is the sum of these atoms:

$$I(X_1, X_2; Y) = R + U_1 + U_2 + S$$

Similarly, the individual mutual information for a single source can be broken down into its unique and redundant contributions:

$$I(X_1; Y) = R + U_1$$ $$I(X_2; Y) = R + U_2$$

By isolating these components, PID allows us to distinguish between systems that rely on backup copies (high redundancy) and systems that compute entirely new properties from combinations of inputs (high synergy).

Resolving the Underdetermined Math

If you look closely at the equations above, a fundamental mathematical problem emerges: we have three classical Shannon equations but four unknown PID atoms ($R$, $U_1$, $U_2$, $S$). The system is underdetermined. To solve this, we must supply an independent, axiomatic definition for at least one of the components—typically the redundant information ($R$) or unique information ($U$).

As Joseph T. Lizier notes in Bits from Brains for Biologically Inspired Computing, defining these boundaries consistently for multivariate systems remains an active area of debate. To ground these definitions rigorously, Gutknecht et al. proposed using the formal logic of part-whole relationships (mereology) in Bits and pieces: understanding information decomposition from part-whole relationships and formal logic, showing that redundant information behaves logically as a shared component of individual source knowledge.

For continuous real-world variables, Ehrlich et al. developed a framework based on "shared exclusions" in probability space, making PID practically estimable via $k$-nearest-neighbor techniques in Partial Information Decomposition for Continuous Variables based on Shared Exclusions: Analytical Formulation and Estimation. Furthermore, a breakthrough detailed in Analytically deriving Partial Information Decomposition for affine systems of stable and convolution-closed distributions proved that closed-form analytical solutions can be derived for non-Gaussian distributions (like Poisson or Cauchy), bypassing computationally expensive numerical optimizations.

Designing Structured Multimodal Fusion

In modern deep learning, multimodal models combine disparate data types—such as text, images, and audio—to make predictions. The standard approach is feature fusion: smashing the representations together via concatenation or simple projection. However, as Yu et al. argue in Explainable Multimodal Regression via Information Decomposition, simple concatenation treats all relationships uniformly, failing to isolate whether modalities provide independent value, overlapping data, or emergent insights.

By integrating PID into neural architectures, models can explicitly handle these distinct streams. For instance, in Leveraging Redundancy and Modality-Specific Features for Robust Multimodal Learning, Nguyen et al. leverage PID to design targeted loss functions. This allows their framework, Robult, to align redundant representations across modalities while strictly preserving unique, modality-specific features—making the model highly resilient to missing sensors or noisy inputs.

Similarly, Xin et al. utilize weakly supervised losses derived from PID in I2MoE: Interpretable Multimodal Interaction-aware Mixture-of-Experts. This forces individual "experts" within a Mixture-of-Experts (MoE) architecture to specialize in either unique, redundant, or synergistic components, providing sample-level explanations of how data sources interact.

Crucially, these interactions are highly dynamic. As Yang et al. demonstrate in Efficient Quantification of Multimodal Interaction at Sample Level, global dataset-level metrics often obscure local dynamics; individual samples within the same category may rely on entirely different information-sharing mechanisms. Their lightweight estimator computes pointwise mutual information to track these shifts at the single-sample level.

Decoding Neural and Latent Representations

PID is equally transformative for understanding biological and artificial neural networks. In neuroscience, the brain must balance redundancy (for error tolerance) and synergy (for complex integration). By applying PID to cortical data, Pica et al. isolated how much sensory information in a neural code is actually relevant for behavioral choices in Quantifying how much sensory information in a neural code is relevant for behavior. In the mouse auditory cortex, Koçillari et al. demonstrated in Behavioural relevance of redundant and synergistic stimulus information between functionally connected neurons in mouse auditory cortex that while both interaction types exist, redundancy serves as the primary driver of correct behavioral choices among functionally connected neurons.

When applied to artificial neural networks, PID provides a rigorous metric for representation complexity and disentanglement. In A Measure of the Complexity of Neural Representations based on Partial Information Decomposition, Schneider et al. show that as networks learn, task-relevant information shifts from complex, synergistic representations across many units to simpler, more accessible redundant or unique representations in deeper layers.

Furthermore, PID helps evaluate whether generative models are truly learning independent factors of variation. In Disentanglement Analysis with Partial Information Decomposition, Tokui et al. use multivariate information decomposition to reveal that certain geometric factors, like orientation, frequently resist one-to-one isolation and instead emerge synergistically across multiple latent variables.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 276 papers
  2. Version 1 · 2026-10-04 · 268 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_partial_information_decomposition_cea8e173,
  title = {partial information decomposition},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/partial-information-decomposition/rdr_cea8e173191b4709861351e2f24e7506},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_partial_information_decomposition_cea8e173_v2,
  title = {partial information decomposition},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/partial-information-decomposition/rdr_cea8e173191b4709861351e2f24e7506/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake