Information-theoretic signatures of causality in Bayesian networks and hypergraphs

Sung En Chiang, Zhaolu Liu, Robert L. Peach, Mauricio Barahona

TL;DR

This paper bridges higher-order information theory and graphical causal models by establishing the first formal correspondence between Partial Information Decomposition (PID) and causal structures. By proving that unique information identifies direct causal neighbors and synergy isolates colliders, the authors introduce a "localist" paradigm that can reconstruct causal networks from a variable's immediate informational footprint without global graph searches.

What Problem Does This Paper Address?

Traditional causal discovery methods (like constraint-based or score-based algorithms) are fundamentally global. Even when they perform local conditional independence tests, a single edge orientation can trigger a cascade of constraint propagations across the entire network to prevent cycles or false v-structures. Furthermore, standard Bayesian networks rely on pairwise edges, making them ill-suited to directly represent higher-order, multi-variable interactions without aggregating information globally.

While higher-order information theory provides tools like Partial Information Decomposition (PID)—which splits the information multiple source variables hold about a target into unique, redundant, and synergistic components—its mathematical connection to causal structure has remained largely undeveloped. This paper addresses this gap by providing a rigorous, model-agnostic foundation that maps PID components directly to structural roles in both standard Bayesian networks and more expressive Bayesian hypergraphs.

Methods

The authors overcome a major bottleneck in PID: as the number of variables grows, the redundancy lattice experiences a combinatorial explosion (scaling with Dedekind numbers). To circumvent this, they employ a bivariate reduction via "supervariables." For any target variable, they group all other variables except one into a single macro-variable, allowing them to exploit well-defined bivariate PID properties while scaling to systems with $d$ variables.

Using this framework, they develop two systematic, step-by-step causal discovery procedures:

  1. For Bayesian Networks: They prove that positive unique information identifies direct causal neighbors (parents or children). They then use synergistic patterns to isolate co-parents (colliders), establishing edge directions locally.
  2. For Bayesian Hypergraphs: They extend this mapping to hypergraphs (which use hyperedges to group multiple tails and heads). Here, unique information characterizes membership in the tail/head or symmetric co-head relations, while synergy isolates co-tail structures. They introduce a Maximal Extension procedure to find the most parsimonious hypergraph representation consistent with the local PID signatures.

Datasets

Not discussed in the paper. This work is purely theoretical and focuses on establishing mathematical proofs and foundational frameworks.

Key Insights and Ideas

  • The Localist Paradigm: Instead of searching the entire graph space or enforcing network-wide consistency, the structure surrounding any variable can be inferred purely from its local informational footprint.
  • Unique Info = Causal Neighbors: A parent exerts an unshareable influence on a child, and a child mirrors its parent uniquely relative to the rest of the system. Thus, unique information acts as a signature for a direct causal edge.
  • Synergy = Colliders: Synergy perfectly captures the "collider effect" (or explaining-away effect), where two independent causes only become dependent when conditioning on a common child.
  • Hypergraphs vs. Pairwise Networks: Bayesian hypergraphs can faithfully represent "partial colliders" (e.g., combinations of logic gates) that standard Bayesian networks distort. As shown in the paper's comparison, conditioning on a child in a standard network falsely induces dependence among all parents, whereas a hypergraph restricts this activation strictly to co-tails within the same hyperedge.
Figure 1: Factorization of distributions according to Bayesian network and Bayesian hyper- graphs. (a) A Bayesian network where X1, X2, X3 and X4 are the parents of X5. (b) A Bayesian hypergraph where X1 and X2 (similarly X3 and X4) are co-parents of X5. Conditioning on X5 only induces dependence between {X1, X2} and {X3, X4} whereas in (a) conditioning on X5 induces dependence among all parent pairs. The factorizations (up to a normalizing constant) are given be- low each structure. Hyperedges are shaded directionally from (co-)tail (light) to (co-)head (dark).
  • Redundancy Reuse: Computing redundant information depends on multiple variables simultaneously, meaning a single redundancy calculation can be reused across different targets to infer multiple unique information terms, drastically lowering computational overhead compared to standard conditional independence testing.

Key Results

  • Theorem 2 & 3 (Bayesian Networks): Proven that $X_j$ has positive unique information about $X_i$ if and only if they are directly connected, and co-parents can be uniquely identified via positive synergy.
  • Theorem 6 (Bayesian Hypergraphs): Proven that PID signatures successfully differentiate parents, children, co-heads, and co-tails in hyperedges, revealing a higher-order collider effect unique to multi-tail hyperedges.
  • Maximal Hyperedge Construction: Defined a mathematical framework to resolve ambiguities where multiple hyperedge configurations match a PID signature, ensuring a unique, canonical graph representation.

Limitations and Future Work

  • Markov Equivalence: Because PID ultimately encodes conditional mutual information, this framework cannot distinguish between causal structures that are observationally Markov-equivalent (e.g., $X \rightarrow Y$ vs. $X \leftarrow Y$ in the absence of colliders). The authors suggest integrating Algorithmic Information Theory (Kolmogorov complexity) in future work to break these symmetries.
  • Estimation Challenges: Computing unique and synergistic information from finite, real-world samples remains notoriously difficult and computationally expensive. Developing reliable, high-dimensional estimation strategies is a critical next step for practical deployment.

Summary updated

Abstract

Analyzing causality in multivariate systems involves establishing how information is generated, distributed and combined. Traditional causal discovery frameworks are capable of multivariate reasoning but their intrinsic pairwise graph topology restricts them to do so only indirectly by integrating multivariate information across pairwise edges. Higher-order information theory provides direct tools that can explicitly model higher-order interactions. In particular, Partial Information Decomposition (PID) allows the decomposition of the information that a set of sources provides about a target into redundant, unique, and synergistic components. Yet the mathematical connection between such higher-order information-theoretic measures and causal structure remains undeveloped. Here we establish the first theoretical correspondence between PID components and causal structure in both Bayesian networks and hypergraphs. We first show that in Bayesian networks unique information precisely characterizes direct causal neighbors, while synergy identifies collider relationships. This establishes a localist causal discovery paradigm in which the structure surrounding each variable can be recovered from its immediate informational footprint, eliminating the need for global search over graph space. Extending these results to more expressive causal representation, we prove that PID signatures in Bayesian hypergraphs differentiate parents, children, co-heads, and co-tails, revealing a novel collider effect unique to multi-tail hyperedges. Our results position PID as a rigorous, model-agnostic foundation for inferring both pairwise and higher-order causal structure, and introduce a fundamentally local information-theoretic viewpoint on causal discovery.
BibTeX
@inproceedings{chiang2026information,
  author = {Sung En Chiang and Zhaolu Liu and Robert L. Peach and Mauricio Barahona},
  title = {{Information-theoretic signatures of causality in Bayesian networks and hypergraphs}},
  booktitle = {CLeaR},
  year = {2026},
  url = {https://openreview.net/forum?id=fjTyhmFZYf},
}
Figure