Estimation and Inference of Heterogeneous Treatment Effects using Random\n Forests

Stefan Wager, Susan Athey

Abstract

Many scientific and engineering challenges—ranging from personalized medicine to customized marketing recommendations—require an understanding of treatment effect heterogeneity. In this article, we develop a nonparametric causal forest for estimating heterogeneous treatment effects that extends Breiman’s widely used random forest algorithm. In the potential outcomes framework with unconfoundedness, we show that causal forests are pointwise consistent for the true treatment effect and have an asymptotically Gaussian and centered sampling distribution. We also discuss a practical method for constructing asymptotic confidence intervals for the true treatment effect that are centered at the causal forest estimates. Our theoretical results rely on a generic Gaussian theory for a large family of random forest algorithms. To our knowledge, this is the first set of results that allows any type of random forest, including classification and regression forests, to be used for provably valid statistical inference. In experiments, we find causal forests to be substantially more powerful than classical methods based on nearest-neighbor matching, especially in the presence of irrelevant covariates.

Introduction

In fields such as personalized medicine, public policy, and economics, the central question is often not whether a treatment works on average, but rather for whom it works best. This is the study of Heterogeneous Treatment Effects (HTE). While standard machine learning (ML) algorithms like Random Forests are exceptionally effective at predicting outcomes, they have historically been treated as "black boxes" regarding statistical uncertainty.

Until recently, it was mathematically difficult to prove that Random Forests follow a standard bell-curve distribution (asymptotic normality), making it nearly impossible to calculate rigorous p-values or 95% confidence intervals. This paper bridges the gap between high-performance machine learning and formal statistical inference by introducing Causal Forests—a method that provides provably valid confidence intervals for treatment effect estimates.

Methodology: The Causal Forest

The authors build upon Leo Breiman’s classic Random Forest algorithm, introducing a critical modification termed "Honesty."

Honest Trees

A tree is defined as "honest" if it uses different subsets of data for different stages of the model-building process. Specifically, the algorithm splits a subsample of the data into two distinct sets:

  1. Set J: Used to determine the structure of the tree (where to place the splits).
  2. Set I: Used to estimate the treatment effects within the resulting leaves.

By decoupling the model structure from the parameter estimation, the authors eliminate the specific type of bias that typically prevents valid statistical inference in adaptive algorithms.

Estimating Treatment Effects

Unlike standard regression trees that predict a mean outcome ($y$), causal trees are designed to estimate the difference between treated and control units ($\tau$) at every leaf. To calculate the associated "error bars," the authors adapt the infinitesimal jackknife technique. This allows for the estimation of the variance of the forest's predictions based on how the output shifts when training observations are slightly re-weighted.

Theoretical Insights

Random Forests as Adaptive Nearest Neighbors

A key conceptual contribution of the paper is the interpretation of a forest as a weighted kernel method. Traditional $k$-Nearest Neighbor ($k$-NN) methods fail in high dimensions because they use a fixed Euclidean distance. In contrast, a Causal Forest "learns" which dimensions are relevant to the treatment effect and defines "closeness" based on those features, effectively acting as an adaptive neighborhood solicitor.

The Necessity of Subsampling

The authors demonstrate that growing trees on small subsamples ($s \ll n$) is a mathematical requirement for inference. This ensures that the bias of the forest decays faster than its variance, allowing the sampling distribution to converge to a Gaussian (normal) distribution. While "honest" trees might appear less efficient because they use only half the data for estimation, the ensemble averaging of the forest compensates for this, often yielding better performance than traditional "dishonest" forests.

Empirical Results

The paper evaluates Causal Forests using simulated datasets designed to mimic difficult real-world conditions, such as high-dimensional noise (up to 30 covariates) and complex non-linear interactions.

  • Statistical Validity: The authors prove that causal forest estimates are pointwise consistent and asymptotically Gaussian. As data volume increases, the estimate converges to the true effect, and errors follow a predictable bell curve.
  • Superiority over k-NN: In scenarios with irrelevant covariates, Causal Forests maintained a low Mean-Squared Error (MSE) of 0.02. In contrast, traditional $k$-NN methods saw error rates explode to 0.33 because they could not distinguish signal from noise.
  • Confidence Interval Coverage: In simulations with 20 dimensions, the 95% confidence intervals produced by the Causal Forest contained the true treatment effect approximately 95% of the time, validating the method's reliability for scientific research.

Limitations and Future Directions

Despite its strengths, the Causal Forest approach has areas for further refinement:

  • Boundary Bias: Like other neighborhood-based methods, Causal Forests can struggle at the edges of the data space. Future work may involve bias-correction techniques for these boundaries.
  • Global Inference: The current theory provides confidence intervals for specific points ($x$). Extending this to "global" confidence bands—inference over the entire function simultaneously—remains an open challenge.
  • Subsample Tuning: While the paper provides a theoretical range for the subsample size ($s$), finding the optimal $s$ for specific empirical datasets is still a matter of practical calibration.

Conclusion

By introducing honesty and leveraging the infinitesimal jackknife, this work transforms Random Forests from a predictive black box into a rigorous tool for causal inference. Causal Forests allow researchers to discover and validate treatment heterogeneity with the same statistical confidence once reserved for simpler linear models, but with the flexibility and power of modern machine learning.

Summary updated

Abstract

Many scientific and engineering challenges -- ranging from personalized\nmedicine to customized marketing recommendations -- require an understanding of\ntreatment effect heterogeneity. In this paper, we develop a non-parametric\ncausal forest for estimating heterogeneous treatment effects that extends\nBreiman's widely used random forest algorithm. In the potential outcomes\nframework with unconfoundedness, we show that causal forests are pointwise\nconsistent for the true treatment effect, and have an asymptotically Gaussian\nand centered sampling distribution. We also discuss a practical method for\nconstructing asymptotic confidence intervals for the true treatment effect that\nare centered at the causal forest estimates. Our theoretical results rely on a\ngeneric Gaussian theory for a large family of random forest algorithms. To our\nknowledge, this is the first set of results that allows any type of random\nforest, including classification and regression forests, to be used for\nprovably valid statistical inference. In experiments, we find causal forests to\nbe substantially more powerful than classical methods based on nearest-neighbor\nmatching, especially in the presence of irrelevant covariates.\n
BibTeX
@article{wager2017estimation,
  author = {Stefan Wager and Susan Athey},
  title = {{Estimation and Inference of Heterogeneous Treatment Effects using Random\textbackslash\{\}n Forests}},
  journal = {Journal of the American Statistical Association},
  year = {2017},
  doi = {10.1080/01621459.2017.1319839},
}
Figure