Estimating Treatment Effects with Causal Forests: An Application

Susan Athey, Stefan WagerObservational 2019 · DOI

TL;DR

This paper applies "causal forests," a machine learning method for estimating heterogeneous treatment effects, to a real-world dataset on student achievement. It highlights the critical importance of accounting for clustered data (students within schools) and demonstrates how causal forests can robustly handle confounding and reveal nuanced treatment effects that simpler methods might miss.

What Problem Does This Paper Address?

In many fields, researchers want to understand the causal impact of an intervention (like a new teaching method) on an outcome (like student achievement). However, in observational studies (where participants aren't randomly assigned to treatment), it's hard to isolate the true effect because of confounding factors. Furthermore, individuals are often grouped (e.g., students in schools), meaning their outcomes might be correlated, which standard statistical methods can struggle to handle. This paper tackles the challenge of estimating heterogeneous treatment effects – how the intervention's impact varies across different individuals or subgroups – in the presence of both confounding and clustered data.

Methods

The core of this paper is the application and discussion of causal forests. This is an extension of the random forest algorithm, originally designed for prediction, to estimate treatment effects. Key methodological contributions and discussions include:

  • Heterogeneous Treatment Effect Estimation: Causal forests aim to estimate the conditional average treatment effect (CATE), $\tau(x) = E[Y(1) - Y(0) | X=x]$, which describes how the treatment effect varies with covariates $x$.
  • Robustness to Confounding: The method uses estimated propensity scores (the probability of receiving treatment given covariates) to adjust for confounding, making the estimates more reliable in observational settings.
  • Handling Clustered Data: A significant contribution is the adaptation of the random forest algorithm to account for clustered observations (students within schools). This involves modifying how data is sampled and how out-of-bag predictions are made to ensure that correlations within clusters are respected. This is crucial for obtaining valid inference.
  • "R-learner" Objective: The causal forest implementation is motivated by an "R-learner" objective function, which balances fitting the treatment effect with controlling for complexity, aiming for efficient estimation of heterogeneous effects.
  • Cross-validation for Tuning: Tuning parameters for the causal forest are selected using cross-validation based on the R-learner objective.

Datasets

The analysis uses a dataset derived from the National Study of Learning Mindsets. This dataset is a simulated version of data from a randomized study conducted in U.S. public high schools, designed to evaluate the impact of a growth mindset intervention on student achievement. The authors note that they are using simulated data for privacy reasons, but that it was fit to the original study's model.

Key Insights and Ideas

  • Clustering is Crucial: The most striking insight is how dramatically ignoring school-level clustering affects the results. Without accounting for it, the analysis falsely suggests strong heterogeneity and provides overly optimistic (narrow) confidence intervals. This underscores the importance of cluster-robust inference in such settings.
  • Causal Forests as a Powerful Tool: The paper demonstrates that causal forests can effectively handle complex data structures (confounding and clustering) and provide nuanced insights into treatment heterogeneity that might be missed by simpler methods.
  • Pre-specified Hypotheses Matter: While general tests for heterogeneity were mixed, focusing on pre-specified school-level covariates (pre-existing mindset and school achievement) revealed more targeted, albeit not always statistically significant, evidence of effect modification.
  • Importance of Nuisance Parameter Estimation: The paper emphasizes that the accuracy of the estimated propensity scores and outcome models (nuisance parameters) is important for the performance of causal forests, especially in observational studies.

Key Results

  • Average Treatment Effect: The average treatment effect of the mindset intervention was estimated to be positive and statistically significant (ATE = 0.247 +/- 0.04).
  • Heterogeneity Findings:
    • General tests for heterogeneity were not statistically significant when using cluster-robust inference, suggesting that any present heterogeneity might be weak or difficult to estimate precisely.
  • A targeted analysis focusing on school-level pre-existing mindset (X1) showed some evidence of negative correlation between X1 and treatment effect, suggesting schools with higher pre-existing mindset might have slightly smaller intervention effects. However, this was not strong enough to pass a Bonferroni correction.
  • No significant heterogeneity was detected based on school-level achievement (X2).
  • Impact of Clustering: When clustering was ignored, the confidence intervals for the ATE were halved, and the analysis falsely detected strong heterogeneity.

Limitations and Future Work

  • Simulated Data: The analysis is based on simulated data, which, while designed to mimic the original study, may not perfectly capture all real-world complexities.
  • Difficulty in Estimating CATE: Even with causal forests, accurately estimating the conditional average treatment effect function $\tau(x)$ proved difficult, particularly when accounting for clustering. The paper suggests that much of the observed school-level variation might be idiosyncratic and not easily explained by covariates.
  • Future Work: The authors suggest that developing a better understanding of different approaches to heterogeneous treatment effect estimation on clustered data is an important area for future research. They also highlight the need for more robust asymptotic results for inference on heterogeneous treatment effects.

Summary updated

Abstract

We apply causal forests to a dataset derived from the National Study of Learning Mindsets, and consider resulting practical and conceptual challenges. In particular, we discuss how causal forests use estimated propensity scores to be more robust to confounding, and how they handle data with clustered errors.
BibTeX
@inproceedings{athey2019estimating,
  author = {Susan Athey and Stefan Wager},
  title = {{Estimating Treatment Effects with Causal Forests: An Application}},
  booktitle = {Observational},
  year = {2019},
  doi = {10.1353/obs.2019.0001},
}
Figure