Bayesian causal inference via probabilistic program synthesis

Sam Witty, Alexander K. Lew, David D. Jensen, Vikash MansinghkaarXiv 2019 · OpenReview

Abstract

Causal inference can be formalized as Bayesian inference that combines a prior distribution over causal models and likelihoods that account for both observations and interventions. We show that it is possible to implement this approach using a sufficiently expressive probabilistic programming language. Priors are represented using probabilistic programs that generate source code in a domain specific language. Interventions are represented using probabilistic programs that edit this source code to modify the original generative process. This approach makes it straightforward to incorporate data from atomic interventions, as well as shift interventions, variance-scaling interventions, and other interventions that modify causal structure. This approach also enables the use of general-purpose inference machinery for probabilistic programs to infer probable causal structures and parameters from data. This abstract describes a prototype of this approach in the Gen probabilistic programming language.

Introduction

Traditional causal inference frameworks often rely on Causal Graphical Models (CGMs) and the "do-operator" to represent interventions. The do-operator typically represents an ideal intervention that fixes a variable to a specific value and removes all incoming causal edges. However, real-world experiments are often more complex, involving "soft" interventions that shift means, scale variances, or modify functional relationships between variables. Jointly inferring causal structure and parameters while accounting for uncertainty in these complex settings remains a significant challenge. This paper introduces a novel framework that treats causal models as programs and interventions as code-editing operations, enabling robust Bayesian causal inference.

Methods: Causal Models as Programs

The proposed framework leverages Bayesian synthesis implemented in the Gen probabilistic programming language. Causal models are specified in a domain-specific language (DSL) called "MiniStan." A high-level Gen program acts as a synthesizer, stochastically generating MiniStan source code to represent a prior over possible causal structures and parameters.

Causal models represented as CGMs and MiniStan source code.

Interventions are modeled as functions that modify the Abstract Syntax Tree (AST) of the MiniStan program. This "code-editing" approach is universal, encompassing standard atomic interventions (replacing a stochastic assignment with a constant) as well as shift interventions (adding a constant to a distribution's mean) and variance-scaling interventions. By running these edited programs through an interpreter, the system calculates the likelihood of both observational and experimental data. Inference is performed using Sequential Monte Carlo (SMC) with Metropolis-Hastings rejuvenation.

Experimental Setup

The framework was evaluated using a conceptual example where a student's success (Outcome) is influenced by their Skill and their Belief in their ability. Synthetic data was generated for 10 individuals across four settings:

  1. Observational: No intervention.
  2. Belief Pill: An atomic intervention fixing Belief to a constant.
  3. Encouragement: A shift intervention increasing the mean of the Belief distribution.
  4. Assessment: A compound intervention that slightly increases Skill while significantly reducing the variance of the Belief distribution.

Results

The results demonstrate the power of combining diverse experimental data for causal discovery. Using only observational data, the posterior probability of the causal edge from Belief to Outcome remained low (~0.3), as simpler models were favored by the Bayesian Ockham's razor. However, as data from various interventions were added, the system correctly identified the true causal structure.

Structure learning and parameter estimation results.

As shown in the results, the posterior probability $P(edge=1|data)$ reached nearly 1.0 when all data sources were combined. Similarly, the posterior distribution for the causal effect parameter $\lambda_{bo}$ concentrated around its true value. The framework's flexibility allowed it to seamlessly handle "imperfect" interventions like encouragement and assessment that are difficult to represent in standard do-calculus.

Conclusion and Future Work

By treating causal models as programs, researchers gain a rich vocabulary for describing real-world experiments. This approach enables the integration of diverse data types into a unified Bayesian inference framework. Future work will focus on improving scalability through advanced inference techniques like Hamiltonian Monte Carlo and extending the framework to non-parametric models and counterfactual reasoning.

Summary updated

Abstract

Causal inference can be formalized as Bayesian inference that combines a prior distribution over causal models and likelihoods that account for both observations and interventions. We show that it is possible to implement this approach using a sufficiently expressive probabilistic programming language. Priors are represented using probabilistic programs that generate source code in a domain specific language. Interventions are represented using probabilistic programs that edit this source code to modify the original generative process. This approach makes it straightforward to incorporate data from atomic interventions, as well as shift interventions, variance-scaling interventions, and other interventions that modify causal structure. This approach also enables the use of general-purpose inference machinery for probabilistic programs to infer probable causal structures and parameters from data. This abstract describes a prototype of this approach in the Gen probabilistic programming language.
BibTeX
@misc{witty2019bayesian,
  author = {Sam Witty and Alexander K. Lew and David D. Jensen and Vikash Mansinghka},
  title = {{Bayesian causal inference via probabilistic program synthesis}},
  year = {2019},
  eprint = {1910.14124},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1910.14124},
}
Figure