An operational information decomposition via synergistic disclosure

Fernando Rosas, Pedro A. M. Mediano, Borzoo Rassouli, Adam B. BarrettApollo 2020 · OpenReview

TL;DR

This paper introduces a novel "synergy-first" approach to decomposing information in complex systems. By operationalizing synergy through a data privacy lens, they provide a clear, computable framework that naturally coarse-grains information, offering a more intuitive understanding of how different parts of a system contribute to predicting a target variable.

What Problem Does This Paper Address?

Understanding complex systems often involves breaking down how information is shared and processed. Multivariate information decompositions aim to quantify this, with a particular interest in "synergy" – information present in the whole system but not in any of its individual parts. However, the field has been plagued by multiple competing decomposition methods and a lack of a clear, operational definition of synergy, making it difficult to interpret and apply these powerful tools. This paper tackles this by proposing a new decomposition that directly defines and quantifies synergy.

Methods

The core of the paper is a new operational definition of synergy, termed "synergistic disclosure." This concept is borrowed from data privacy literature and quantifies the information about a target variable Y that can be revealed without disclosing information about specific subsystems of the sources X. This is achieved by defining "synergistic channels" that mask information about subsystems.

Using this definition, the authors build a multivariate information decomposition based on the Möbius inversion formula applied to a specially constructed "extended constraint lattice." This lattice orders different combinations of source variables based on their synergistic disclosure properties. The decomposition yields "synergy atoms" that represent the unique contribution of each combination of sources to predicting Y.

A key innovation is the "backbone decomposition," which provides a natural, linear-time coarse-graining of the full decomposition, making it tractable for larger systems. This backbone decomposition consists of "backbone atoms" that quantify information disclosure under progressively relaxed constraints.

Datasets

Not discussed in the paper. The paper focuses on theoretical development and uses synthetic examples and simulations based on probability distributions (e.g., AND gate, XOR, copy, Ising models) rather than real-world datasets.

Key Insights and Ideas

  • Synergy as Disclosure: The paper reframes synergy not just as a statistical property, but as information that can be disclosed about a target without revealing information about its constituent parts. This operational definition is a significant step towards interpretability.
  • Data Privacy as a Tool: Leveraging concepts from data privacy provides a concrete, computable way to define and measure synergy, moving beyond purely axiomatic approaches.
  • Backbone Decomposition for Scalability: The backbone decomposition offers a practical way to summarize complex information structures by focusing on a linearly growing set of atoms, making the approach scalable to larger systems.
  • Synergy is Prevalent: The authors' work suggests that high-order synergies are not rare corner cases but are common in probability distributions, particularly in self-disclosure scenarios.
  • "Synergy-First" Approach: Unlike many previous methods that derive synergy as a byproduct, this paper builds the entire decomposition around the concept of synergy, offering a potentially more coherent view.

Key Results

  • The paper introduces a novel information decomposition based on synergistic disclosure, applicable to any number of source variables.
  • The decomposition's atoms can be calculated using elementary optimization techniques.
  • A "backbone decomposition" is proposed, providing a natural coarse-graining that scales linearly with system size.
  • Numerical examples demonstrate that the proposed synergy measure is generally more conservative than other PID measures, attributing smaller values to synergy.
  • The framework offers clear interpretations for challenging problems like the "two-bit-copy" problem.
  • The authors show that high-order synergies are prevalent, with backbone atoms often capturing most of the information as system size grows.

Limitations and Future Work

  • Comparison with other PIDs: While the paper shows correlations between its synergy measure and others, and highlights differences in how "restrictive" they are, a deeper understanding of the precise relationships and potential equivalences remains an area for future work.
  • Generalization of Self-Synergy: While the paper shows that the optimal self-synergistic channel for binary variables can be a sufficient statistic, numerical explorations suggest this doesn't hold in general, indicating further research is needed.
  • Analytical Connections: The paper notes that finding analytical methods to connect Hamiltonian interaction orders with backbone atoms is an open question.
  • Practical Applications: The authors suggest their framework opens doors for applications in data analysis, neuroscience, and information dynamics, implying that exploring these specific applications is a future direction.
  • Software Availability: An open-source software package is provided, facilitating future research and applications.

Summary updated

Abstract

Multivariate information decompositions hold promise to yield insight into complex systems, and stand out for their ability to identify synergistic phenomena. However, the adoption of these approaches has been hindered by there being multiple possible decompositions, and no precise guidance for preferring one over the others. At the heart of this disagreement lies the absence of a clear operational interpretation of what synergistic information is. Here we fill this gap by proposing a new information decomposition based on a novel operationalisation of informational synergy, which leverages recent developments in the literature of data privacy. Our decomposition is defined for any number of information sources, and its atoms can be calculated using elementary optimisation techniques. The decomposition provides a natural coarse-graining that scales gracefully with the system's size, and is applicable in a wide range of scenarios of practical interest.
BibTeX
@inproceedings{rosas2020operational,
  author = {Fernando Rosas and Pedro A. M. Mediano and Borzoo Rassouli and Adam B. Barrett},
  title = {{An operational information decomposition via synergistic disclosure}},
  booktitle = {Apollo},
  year = {2020},
  url = {https://openreview.net/forum?id=CzVMQgzifB},
}
Figure