Estimating causal effects with optimization-based methods: A review and empirical comparison

Martin Cousineau, Vedat Verter, Susan A. Murphy, Joelle Pineau

Introduction

In the quest to determine the effectiveness of interventions—ranging from new medical treatments to public policy changes—Randomized Controlled Trials (RCTs) remain the gold standard. However, in many real-world scenarios, RCTs are impractical, unethical, or prohibitively expensive. Researchers are instead forced to rely on observational data, which is inherently plagued by confounding: the risk that the groups being compared are fundamentally different in ways that influence the outcome.

This paper provides a comprehensive review and benchmarking of a modern class of causal inference techniques: optimization-based balancing methods. These methods leverage mathematical programming to create comparable groups, offering a more transparent and controlled alternative to traditional statistical approaches.

The Three Paradigms of Causal Inference

The authors categorize the landscape of causal effect estimation into three primary methodological buckets:

1. Balancing Methods

These methods focus exclusively on the "design" stage of a study. The goal is to create weights or select subgroups such that the distribution of covariates (like age, income, or health status) is identical between the treatment and control groups. By doing this before ever looking at the outcome data, researchers can prevent "p-hacking" and ensure an objective comparison.

Workflow of a balancing-based causal inference method, focusing on the design stage to eliminate bias before outcome analysis.

2. Regression Methods

Rather than focusing on group balance, regression methods model the "response surface"—the direct relationship between covariates and outcomes. By mathematically accounting for the influence of confounding variables, these models attempt to isolate the specific effect of the treatment.

Workflow of a regression-based method, which models the relationship between covariates and outcomes directly to estimate effects.

3. Mixed Methods

Mixed methods attempt to get the best of both worlds. They use optimization to achieve initial balance and then apply regression models to the balanced data to further refine the estimate and reduce residual bias.

Optimization-Based Balancing Techniques

The core contribution of the paper is its deep dive into optimization-based balancing. Unlike traditional propensity score matching, which often feels like a "black box" of trial and error, optimization methods allow researchers to specify exact constraints. Key methods reviewed include:

  • MIPMatch: Employs Mixed-Integer Programming to find the mathematically optimal matches between subjects.
  • Entropy Balancing (EBal): Finds weights that satisfy specific moment constraints (e.g., ensuring the mean age is identical across groups) while staying as close as possible to uniform weights.
  • Kernel Optimal Matching (KBal): Uses "kernel tricks" to balance an infinite number of moments in a high-dimensional space, capturing complex nonlinear relationships.
  • Approximate Residual Balancing (ARB): A hybrid approach combining optimization-based weights with penalized Lasso regression.

Empirical Benchmarking: The ACIC 2016 Challenge

To test these methods, the authors conducted a massive empirical comparison using 7,700 unique datasets from the 2016 Atlantic Causal Inference Conference (ACIC) competition. These datasets are based on real-world covariates from the Collaborative Perinatal Project (4,802 subjects, 58 variables), but with simulated treatment assignments and outcomes. This setup allows researchers to know the "ground truth" causal effect while testing methods against the complexities of real-world data.

Key Findings and the "Performance Gap"

Despite the mathematical elegance and transparency of optimization methods, the study revealed a significant performance gap:

  • The Dominance of BART: Bayesian Additive Regression Trees (BART), a flexible machine learning regression method, significantly outperformed all pure optimization-based balancing methods in terms of Root Mean Square Error (RMSE) and bias.
  • Leaderboard Standings: Most optimization-based methods landed in the bottom 25% of the 24 methods compared. While they are more transparent than traditional matching, they currently struggle to match the raw predictive power of advanced regression models.
  • Robustness Issues: Popular methods like Entropy Balancing (EBal) failed to produce results on nearly 40% of the datasets due to variable collinearity, highlighting a critical need for more robust algorithmic implementations.
  • Computational Efficiency: Speed varied wildly. EBal was lightning-fast (0.2 seconds), while more complex methods like KBal and GenMatch took over an hour per dataset, limiting their utility for large-scale applications.

The Path Forward

The authors conclude that while optimization methods offer unprecedented control during the study design phase, they are not yet a "silver bullet" for accuracy. The most promising future direction lies in Optimization-based Mixed Methods—using mathematical programming to ensure robust design and then feeding that data into high-powered models like BART for the final analysis. Additionally, the Operational Research community has a significant opportunity to improve the scalability and tuning of these algorithms to make them more competitive in high-stakes data environments.

Summary updated

BibTeX
@article{cousineau2023estimating,
  author = {Martin Cousineau and Vedat Verter and Susan A. Murphy and Joelle Pineau},
  title = {{Estimating causal effects with optimization-based methods: A review and empirical comparison}},
  journal = {Eur. J. Oper. Res.},
  year = {2023},
  doi = {10.1016/j.ejor.2022.01.046},
}
Figure