Model Criticism for Bayesian Causal Inference

Dustin Tran, Francisco J. R. Ruiz, Susan Athey, David M. BleiarXiv 2016 · DOI · arXiv

Abstract

The goal of causal inference is to understand the outcome of alternative courses of action. However, all causal inference requires assumptions. Such assumptions can be more influential than in typical tasks for probabilistic modeling, and testing those assumptions is important to assess the validity of causal inference. We develop model criticism for Bayesian causal inference, building on the idea of posterior predictive checks to assess model fit. Our approach involves decomposing the problem, separately criticizing the model of treatment assignments and the model of outcomes. Conditioned on the assumption of unconfoundedness---that the treatments are assigned independently of the potential outcomes---we show how to check any additional modeling assumption. Our approach provides a foundation for diagnosing model-based causal inferences.

Introduction

Causal inference is the science of "what if." To estimate a causal effect—such as the impact of a new medical treatment—researchers must rely on heavy assumptions because we never observe the counterfactual: what would have happened if a patient had received a different treatment. While Bayesian frameworks allow for complex modeling of these effects, they have historically lacked robust tools to verify if the models are accurately capturing reality.

The "Fundamental Problem of Causal Inference" is that for any individual, we only observe one outcome. Standard Bayesian "goodness-of-fit" tests, known as Posterior Predictive Checks (PPCs), typically require complete data. In causal settings, attempting to "fill in" missing counterfactuals with a model's own predictions creates a circular logic trap where the model merely confirms its own biases. This paper introduces a diagnostic framework that avoids this pitfall through model decomposition and reweighting.

Methodology: A Two-Part Diagnostic Framework

The core innovation of this framework is the decomposition of the causal model into two distinct components: the Assignment Model and the Outcome Model. This separation allows researchers to pinpoint exactly where a model might be failing.

1. Assignment Model Criticism

The assignment model predicts the probability of an individual receiving a specific treatment. This component can be evaluated using standard PPCs by comparing actual treatment assignments to replicated assignments generated by the model. If the distributions differ significantly, the assignment mechanism is likely misspecified.

Assignment model criticism demonstrating a clear failure (tail prob = 1.00) when the assignment mechanism is incorrectly modeled.

2. Outcome Model Criticism via Importance Reweighting

Criticizing the outcome model is more challenging due to the lack of observed counterfactuals. The authors demonstrate that simple imputation leads to biased diagnostics. Instead, they propose using propensity scores (the predicted probability of treatment) to reweight the observed data. This "realized discrepancy" approach allows for checking the outcome model's fit against the observed data distribution without relying on the model's own potentially flawed predictions for missing data.

Outcome model diagnostics showing sensitivity to outcome misspecification (b) while remaining robust to assignment errors (c), highlighting the precision of the decomposition approach.

Key Insights

  • The Imputation Trap: The authors mathematically prove that imputing missing counterfactuals to check a model leads to overly optimistic, biased results. It makes the model appear to fit better than it does because the "ground truth" is derived from the model's own opinion.
  • Divide and Conquer: By separating the assignment and outcome models, researchers can identify if an issue stems from how subjects are selected for treatment or how the treatment itself works. As shown in the figures above, the outcome check is specifically sensitive to outcome errors and robust to assignment errors.

Empirical Evaluation

The framework was validated across several datasets:

  • Synthetic Data: Confirmed that the method detects deliberate misspecifications, such as omitted variables, that standard checks miss.
  • Cockroach Infestation Study: Revealed overdispersion in a Poisson outcome model, guiding the researchers toward a more accurate model where variance is a linear function of the mean.
  • The Electric Company Study: Showed that simplistic models failed to capture grade-specific nuances in educational outcomes from a randomized experiment.

Conclusion

This work provides a principled foundation for diagnosing Bayesian causal models. By avoiding the circularity of counterfactual imputation and leveraging model decomposition, it offers a transparent way to test the modeling assumptions that underpin causal claims. While the fundamental assumption of "unconfoundedness" remains untestable, this toolkit ensures that all other components of the model are held to a rigorous standard of criticism.

Summary updated

Abstract

The goal of causal inference is to understand the outcome of alternative courses of action. However, all causal inference requires assumptions. Such assumptions can be more influential than in typical tasks for probabilistic modeling, and testing those assumptions is important to assess the validity of causal inference. We develop model criticism for Bayesian causal inference, building on the idea of posterior predictive checks to assess model fit. Our approach involves decomposing the problem, separately criticizing the model of treatment assignments and the model of outcomes. Conditioned on the assumption of unconfoundedness---that the treatments are assigned independently of the potential outcomes---we show how to check any additional modeling assumption. Our approach provides a foundation for diagnosing model-based causal inferences.
BibTeX
@misc{tran2016model,
  author = {Dustin Tran and Francisco J. R. Ruiz and Susan Athey and David M. Blei},
  title = {{Model Criticism for Bayesian Causal Inference}},
  year = {2016},
  eprint = {1610.09037},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1610.09037},
}
Figure