Causal version of Principle of Insufficient Reason and MaxEnt

Dominik JanzingarXiv 2021 · OpenReview

TL;DR

This paper introduces "Causal PIR" and "Causal MaxEnt," novel principles that address paradoxical probability updates in cause-effect scenarios. By separating constraints into those on the cause and those on the mechanism linking cause and effect, the authors provide a principled justification for sequential entropy maximization, resolving issues with standard approaches and offering a new perspective on causal inference.

What Problem Does This Paper Address?

When inferring cause-effect relationships from data, standard probabilistic principles like the Principle of Insufficient Reason (PIR) and Maximum Entropy (MaxEnt) can lead to counterintuitive results. Specifically, if we know the conditional probability of an effect given a cause, but not the cause's prior distribution, standard MaxEnt might assign higher probability to causes that offer more possible outcomes for the effect. This "intentional behavior" is problematic because it suggests the cause "knows" about the effect's possibilities beforehand. Previous attempts to fix this involved sequentially applying MaxEnt according to the causal order, but lacked a strong theoretical foundation. This paper aims to provide that foundation by reformulating PIR and MaxEnt in a causal context.

Methods

The core contribution is the introduction of "Causal PIR" and "Causal MaxEnt." Causal PIR is a modification of the standard PIR that accounts for the causal direction. It suggests assigning uniform probabilities to possible causes given the observed effect, but with a crucial asymmetry: it prioritizes causes that are "simpler" or have fewer options for the effect. This is justified by Postulate 2, which proposes separating constraints into those that restrict the cause and those that restrict the mechanism generating the effect. Causal MaxEnt is then derived from Causal PIR by applying sequential maximization of conditional entropy, first for the cause and then for the effect given the cause. The paper also shows how Information Geometric Causal Inference (IGCI) can be derived from Causal PIR.

Datasets

Not discussed in the paper. The paper focuses on theoretical principles and uses toy examples (like a mechanical device with channels) to illustrate its concepts.

Key Insights and Ideas

  • Causal Asymmetry is Key: The fundamental insight is that cause-effect relationships are not symmetric. Standard probabilistic principles, which are often symmetric, fail when applied naively to causal scenarios.
  • Separating Constraints: The paper's main technical contribution is the idea of separating constraints into those that apply to the cause itself and those that apply to the mechanism linking cause and effect. This separation is crucial for avoiding paradoxical updates.
  • Justifying Sequential Updates: The proposed "Causal PIR" and "Causal MaxEnt" provide a principled justification for sequentially updating probabilities according to the causal order, resolving the "intentional behavior" paradox.
  • Information Geometric Causal Inference as a Consequence: The paper demonstrates that a known causal discovery method, IGCI, can be derived from the proposed Causal PIR, suggesting a deeper connection between these approaches.
  • PIR as a Foundation for MaxEnt: The paper argues that Causal PIR can serve as a more fundamental principle from which Causal MaxEnt can be derived, offering a new perspective on the relationship between these two concepts.

Key Results

  • The paper introduces "Causal PIR" and "Causal MaxEnt" as principled ways to handle probability updates in cause-effect scenarios.
  • It provides a theoretical justification for sequentially maximizing entropy according to the causal order, resolving paradoxes associated with standard MaxEnt.
  • It shows that Information Geometric Causal Inference (IGCI) can be derived from Causal PIR.
  • The paper highlights potential issues with generalizing Causal MaxEnt to arbitrary causal DAGs, particularly when constraints are not "tight."

Limitations and Future Work

  • Generalization to Arbitrary DAGs: The paper briefly discusses challenges in generalizing Causal MaxEnt to arbitrary causal Directed Acyclic Graphs (DAGs), especially when constraints are not "tight" (meaning they don't fully determine the relationship). This remains an open area.
  • Ontic vs. Epistemic Interpretation: While the paper touches upon the distinction between constraints arising from mechanisms (ontic) versus those from knowledge (epistemic), a deeper exploration of this is suggested.
  • Practical Implementation: The paper focuses on theoretical justification and derivation, with proposing new practical inference rules being beyond its scope.
  • Justification of Postulate 2: While Postulate 2 (separation of constraints) is central, its justification is presented as informal and related to "abstract physics" and thermodynamic arrows of time, suggesting further theoretical work could strengthen it.

Summary updated

Abstract

The Principle of Insufficient Reason (PIR) assigns equal probabilities to each alternative of a random experiment whenever there is no reason to prefer one over the other. The Maximum Entropy Principle (MaxEnt) generalizes PIR to the case where statistical information like expectations are given. It is known that both principles result in paradoxical probability updates for joint distributions of cause and effect. This is because constraints on the conditional P(effect|cause) result in changes of P(cause) that assign higher probability to those values of the cause that offer more options for the effect, suggesting "intentional behaviour". Earlier work therefore suggested sequentially maximizing (conditional) entropy according to the causal order, but without further justification apart from plausibility on toy examples. We justify causal modifications of PIR and MaxEnt by separating constraints into restrictions for the cause and restrictions for the mechanism that generates the effect from the cause. We further sketch why Causal PIR also entails "Information Geometric Causal Inference". We briefly discuss problems of generalizing the causal version of MaxEnt to arbitrary causal DAGs.
BibTeX
@misc{janzing2021causal,
  author = {Dominik Janzing},
  title = {{Causal version of Principle of Insufficient Reason and MaxEnt}},
  year = {2021},
  eprint = {2102.03906},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2102.03906},
}
Figure