Synth-Validation: Selecting the Best Causal Inference Method for a Given\n Dataset

Alejandro Schuler, Ken Jung, Robert Tibshirani, Trevor Hastie, Nigam H. ShaharXiv 2017 · DOI · arXiv

Authors: Ken Jung, Robert Tibshirani, Trevor Hastie, Nigam Shah

Abstract: Many decisions in healthcare, business, and other policy domains are made without the support of rigorous evidence due to the cost and complexity of performing randomized experiments. Using observational data to answer causal questions is risky: subjects who receive different treatments also differ in other ways that affect outcomes. Many causal inference methods have been developed to mitigate these biases. However, there is no way to know which method might produce the best estimate of a treatment effect in a given study. In analogy to cross-validation, which estimates the prediction error of predictive models applied to a given dataset, we propose synth-validation, a procedure that estimates the estimation error of causal inference methods applied to a given dataset. In synth-validation, we use the observed data to estimate generative distributions with known treatment effects. We apply each causal inference method to datasets sampled from these distributions and compare the effect estimates with the known effects to estimate error. Using simulations, we show that using synth-validation to select a causal inference method for each study lowers the expected estimation error relative to consistently using any single method.

Introduction

In fields like healthcare and public policy, randomized controlled trials are often prohibitively expensive or unethical. Consequently, researchers rely on observational data to estimate causal effects. However, observational studies are plagued by confounding: individuals receiving different treatments often differ in ways that independently affect outcomes. While numerous causal inference methods—ranging from propensity score matching to machine learning-based estimators—exist to mitigate these biases, there is no universal "best" method. The performance of any given estimator depends heavily on the underlying data distribution, which is unknown in practice.

In predictive modeling, cross-validation provides a robust, data-driven way to select models by comparing predictions to observed outcomes. In causal inference, this is impossible because of the "Fundamental Problem": we never observe the counterfactual. We see what happened to a patient who took a drug, but we can never see what would have happened to that same patient if they had not. This paper introduces synth-validation, a procedure that brings the empirical selection logic of cross-validation to the causal domain.

The Synth-Validation Procedure

Synth-validation estimates the estimation error of various causal inference methods by applying them to synthetic datasets where the ground truth is known. The workflow consists of four primary stages:

  1. Estimation of Plausible Effects: The researcher first applies several standard causal methods to the original dataset to identify a range of "plausible" treatment effects.
  2. Generative Modeling with CGB-tree: The core innovation is the use of Constrained Gradient Boosting (CGB-tree). This algorithm fits a generative model to the observed data while enforcing a specific, pre-defined treatment effect. It ensures the synthetic data remains as realistic as possible while providing a known "true" effect for benchmarking.
  3. Benchmarking: Multiple synthetic datasets are generated. Candidate causal methods are then applied to these datasets.
  4. Selection: The estimation error for each method is calculated by comparing its estimates against the known synthetic effects. The method with the lowest average error is selected for use on the original dataset.
The synth-validation workflow: from observed data to synthetic benchmarking and final method selection.

Experimental Setup and Results

The authors validated the procedure using 16 simulation scenarios (8 randomized, 8 observational) comprising 160 distinct datasets. They compared five common estimators: raw difference in means, covariate-adjusted linear regression, 1:1 propensity score matching, Inverse Probability of Treatment Weighting (IPTW), and a machine-learning "Boosting" estimator.

Key Results

  • Lower Error: Using synth-validation to select a method resulted in lower expected estimation error compared to consistently using any single causal method across all scenarios.
  • Matching the Oracle: In many scenarios, synth-validation's performance was nearly indistinguishable from an "Oracle" selector (a theoretical selector that knows the true effect and always picks the best method).
  • Robustness: The procedure proved remarkably robust; the choice of the "plausible effect" range in the first step did not significantly impact the final selection accuracy.

The Role of CGB-tree

The CGB-tree algorithm is critical to the success of synth-validation. Unlike simpler methods that might add a constant effect to all outcomes, CGB-tree adjusts outcomes more heavily in regions with low data support. This mimics the subtle ways confounding often manifests in real-world data, making the synthetic benchmarks more challenging and realistic.

Limitations and Future Directions

While synth-validation offers a principled approach to method selection, it does not solve the problem of unmeasured confounding—if a critical variable is missing from the data, no selection method can recover the true effect. Future work may involve simulating unmeasured confounding by withholding variables during the synthetic generation process. Additionally, while the current work focuses on continuous outcomes, the framework can be extended to binary or survival data through appropriate link functions.

Summary updated

Abstract

Many decisions in healthcare, business, and other policy domains are made\nwithout the support of rigorous evidence due to the cost and complexity of\nperforming randomized experiments. Using observational data to answer causal\nquestions is risky: subjects who receive different treatments also differ in\nother ways that affect outcomes. Many causal inference methods have been\ndeveloped to mitigate these biases. However, there is no way to know which\nmethod might produce the best estimate of a treatment effect in a given study.\nIn analogy to cross-validation, which estimates the prediction error of\npredictive models applied to a given dataset, we propose synth-validation, a\nprocedure that estimates the estimation error of causal inference methods\napplied to a given dataset. In synth-validation, we use the observed data to\nestimate generative distributions with known treatment effects. We apply each\ncausal inference method to datasets sampled from these distributions and\ncompare the effect estimates with the known effects to estimate error. Using\nsimulations, we show that using synth-validation to select a causal inference\nmethod for each study lowers the expected estimation error relative to\nconsistently using any single method.\n
BibTeX
@misc{schuler2017synth,
  author = {Alejandro Schuler and Ken Jung and Robert Tibshirani and Trevor Hastie and Nigam H. Shah},
  title = {{Synth-Validation: Selecting the Best Causal Inference Method for a Given\textbackslash\{\}n Dataset}},
  year = {2017},
  eprint = {1711.00083},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1711.00083},
}
Figure