High-Dimensional Feature Selection for Sample Efficient Treatment Effect Estimation

Kristjan H. Greenewald, Dmitriy A. Katz-Rogozhnikov, Karthik Shanmugam

Introduction

Causal inference often faces the "curse of dimensionality." To estimate the Average Treatment Effect (ATE) without bias, researchers must adjust for numerous covariates. However, as the number of variables ($p$) grows, the required sample size ($n$) typically explodes. This paper, from MIT-IBM Watson AI Lab and IBM Research, proposes a feature selection method that identifies the minimal sufficient subset of covariates, enabling accurate estimation even in high-dimensional settings with limited samples.

Identifying the Optimal Feature Subset

The core innovation lies in identifying a subset $S$ that includes not just confounders (variables affecting both treatment $T$ and outcome $Y$), but also predictors (variables affecting only $Y$). While confounders are necessary for bias removal, predictors are crucial for reducing the variance of the estimate. By including outcome predictors ($X_3$), the model cleans up "noise" in the outcome variable, making the treatment effect more discernible.

Causal graph illustrating the subset $S$ of features ($X_1, X_2, X_3$) relevant for treatment effect estimation.

The authors utilize joint sparse non-convex regularization (e.g., SCAD or MCP). By coupling features across treatment groups, the model "borrows strength" across cohorts, assuming that relevant features for one group are likely relevant for others. This multi-task learning approach makes the selection process much more robust than independent feature selection for each treatment level.

Theoretical and Empirical Results

The proposed algorithm achieves a sample complexity of $O(kq \log p)$, where $k$ is the number of relevant features and $q$ is the number of treatment levels. This logarithmic scaling with respect to $p$ is a significant theoretical advancement, implying that increasing the number of irrelevant variables requires only a minimal increase in sample size to maintain accuracy.

Support recovery rate as a function of sample size $n$ for varying feature dimensions $p$, demonstrating logarithmic scaling.

Empirical validation on synthetic data confirms this scaling. On real-world datasets like Cattaneo2 (maternal smoking and birth weight) and IDHP (infant health), the sparse method outperformed traditional non-sparse estimators. For instance, on IDHP, the method achieved an ATE of 4.49 (close to the true 4.36) using an average of only 6.4 features out of 25, outperforming standard "doubly robust" estimators which are often considered the gold standard.

Conclusion

By focusing on joint sparsity and non-convex regularization, this work provides a robust framework for high-dimensional causal inference. While currently focused on linear models and discrete treatments, future work aims to extend these guarantees to non-linear models (such as neural networks) and continuous treatment regimes.

Summary updated

BibTeX
@inproceedings{greenewald2021high,
  author = {Kristjan H. Greenewald and Dmitriy A. Katz-Rogozhnikov and Karthik Shanmugam},
  title = {{High-Dimensional Feature Selection for Sample Efficient Treatment Effect Estimation}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2021},
  url = {https://openreview.net/forum?id=96FiZN0taxN},
}
Figure