Calibrated Propensity Scores for Causal Effect Estimation

Shachi Deshpande, Volodymyr Kuleshov

Abstract

Propensity scores are commonly used to estimate treatment effects from observational data. We argue that the probabilistic output of a learned propensity score model should be calibrated -- i.e., a predictive treatment probability of 90% should correspond to 90% of individuals being assigned the treatment group -- and we propose simple recalibration techniques to ensure this property. We prove that calibration is a necessary condition for unbiased treatment effect estimation when using popular inverse propensity weighted and doubly robust estimators. We derive error bounds on causal effect estimates that directly relate to the quality of uncertainties provided by the probabilistic propensity score model and show that calibration strictly improves this error bound while also avoiding extreme propensity weights. We demonstrate improved causal effect estimation with calibrated propensity scores in several tasks including high-dimensional image covariates and genome-wide association studies (GWASs). Calibrated propensity scores improve the speed of GWAS analysis by more than two-fold by enabling the use of simpler models that are faster to train.

Introduction

In observational studies where treatment is not randomly assigned, researchers use propensity scores—the probability of receiving treatment given specific characteristics—to correct for confounding bias. A standard tool for this is Inverse Propensity Treatment Weighting (IPTW). However, modern machine learning models, such as neural networks, are often "uncalibrated," meaning their predicted probabilities do not match observed frequencies. For instance, an overconfident model might predict a 0.1% chance of treatment when the true probability is 10%. When inverted ($1/0.001 = 1000$), these incorrect probabilities cause weights to explode, leading to high variance and biased results. While "trimming" extreme weights is a common fix, it often introduces new biases. This research proposes fixing the underlying uncertainty of the model through calibration instead.

Methodology

We propose a two-step framework for causal estimation:

  1. Base Model Training: Use any standard classifier (e.g., Logistic Regression, MLP, or Naive Bayes) to estimate initial propensity scores.
  2. Recalibration: Apply post-hoc recalibration techniques, such as Platt Scaling or Isotonic Regression, using a hold-out calibration set.

Theoretical analysis proves that calibration is a necessary condition for unbiased treatment effect estimation when using IPTW and Doubly Robust (AIPW) estimators. We also derive error bounds that directly link the quality of a model's uncertainty to the accuracy of the final causal estimate.

Experimental Evaluation

The approach was evaluated across three distinct settings:

  • Drug Effectiveness Simulation: A synthetic study where treatment assignment followed complex logical rules (AND/XOR conditions).
  • Unstructured Image Covariates: A benchmark using MNIST images as high-dimensional confounders.
  • Genome-Wide Association Studies (GWAS): Simulated genetic datasets (Spatial, TGP, HGDP) involving thousands of individuals and genetic markers (SNPs).

Key Results and Discussion

Across all simulations, recalibration consistently reduced the Absolute Treatment Effect (ATE) error compared to uncalibrated models. Calibrated models are mathematically discouraged from producing the extreme probabilities that cause IPTW weights to explode, providing natural variance control without the need for ad-hoc trimming.

Density of treatment effect estimates across different methods. Calibrated propensities (green) align more closely with the RCT baseline (blue) than uncalibrated ones (red).

In genomic studies, the use of calibrated simple models yielded significant efficiency gains. A calibrated Naive Bayes model matched the accuracy of state-of-the-art, computationally expensive Linear Mixed Models (LMMs) while achieving a two-fold increase in processing speed.

Comparison of ATE error and throughput (SNPs/sec) for GWAS analysis. Calibrated Naive Bayes (NB) achieves lower error than LMM while maintaining high throughput.

Conclusion

Calibration is not merely a diagnostic tool but a critical requirement for accurate causal inference. By ensuring propensity models are well-calibrated, researchers can reduce estimation error and leverage simpler, faster models for complex tasks like GWAS. Future work will explore extending these principles to continuous treatments and more advanced conformal prediction methods.

Summary updated

Abstract

Propensity scores are commonly used to estimate treatment effects from observational data. We argue that the probabilistic output of a learned propensity score model should be calibrated -- i.e., a predictive treatment probability of 90% should correspond to 90% of individuals being assigned the treatment group -- and we propose simple recalibration techniques to ensure this property. We prove that calibration is a necessary condition for unbiased treatment effect estimation when using popular inverse propensity weighted and doubly robust estimators. We derive error bounds on causal effect estimates that directly relate to the quality of uncertainties provided by the probabilistic propensity score model and show that calibration strictly improves this error bound while also avoiding extreme propensity weights. We demonstrate improved causal effect estimation with calibrated propensity scores in several tasks including high-dimensional image covariates and genome-wide association studies (GWASs). Calibrated propensity scores improve the speed of GWAS analysis by more than two-fold by enabling the use of simpler models that are faster to train.
BibTeX
@inproceedings{deshpande2024calibrated,
  author = {Shachi Deshpande and Volodymyr Kuleshov},
  title = {{Calibrated Propensity Scores for Causal Effect Estimation}},
  booktitle = {Conference on Uncertainty in Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=xgs9VpvVAC},
}
Figure