A Distillation Approach to Data Efficient Individual Treatment Effect Estimation

Maggie Makar, Adith Swaminathan, Emre KicimanAAAI 2019 · OpenReview

Abstract

The potential for using machine learning algorithms as a tool for suggesting optimal interventions has fueled significant interest in developing methods for estimating heterogeneous or individual treatment effects (ITEs) from observational data. While several methods for estimating ITEs have been recently suggested, these methods assume no constraints on the availability of data at the time of deployment or test time. This assumption is unrealistic in settings where data acquisition is a significant part of the analysis pipeline, meaning data about a test case has to be collected in order to predict the ITE. In this work, we present Data Efficient Individual Treatment Effect Estimation (DEITEE), a method which exploits the idea that adjusting for confounding, and hence collecting information about confounders, is not necessary at test time. DEITEE allows the development of rich models that exploit all variables at train time but identifies a minimal set of variables required to estimate the ITE at test time. Using 77 semi-synthetic datasets with varying data generating processes, we show that DEITEE achieves significant reductions in the number of variables required at test time with little to no loss in accuracy. Using real data, we demonstrate the utility of our approach in helping soon-to-be mothers make planning and lifestyle decisions that will impact newborn health.

Introduction

Estimating the Individual Treatment Effect (ITE)—the specific benefit an intervention provides to a particular individual—is a central challenge in causal inference. To obtain unbiased estimates from observational data, researchers must typically account for a wide array of confounders—variables that influence both the treatment assignment and the outcome. However, existing machine learning models for ITE generally assume that all variables used during training will be available during deployment. This is often impractical in real-world scenarios, such as emergency medical settings, where data acquisition is costly or time-consuming. This paper introduces DEITEE (Data Efficient Individual Treatment Effect Estimation) to bridge the gap between high-fidelity causal discovery and efficient real-time prediction.

Methodology: The DEITEE Approach

DEITEE employs a two-stage distillation process to separate the discovery of causal effects from their efficient prediction:

  1. ITE Discovery (Base Model): A high-capacity model, such as Bayesian Additive Regression Trees (BART) or Generalized Random Forests (GRF), is trained on the full set of available data, including all potential confounders. This stage ensures the causal surface is learned accurately by adjusting for selection bias.
  2. ITE Prediction (Distilled Model): The knowledge from the base model is distilled into a lightweight decision tree. Unlike standard models, this tree is trained to predict the ITE values estimated by the base model rather than the raw outcomes.
A distilled decision tree illustrates how DEITEE prioritizes questions to estimate treatment effects efficiently.

The choice of a decision tree provides several advantages. It allows for personalized querying, where the sequence of questions asked is tailored to the individual's previous answers. Furthermore, it enables early estimation, providing a valid ITE estimate at any depth of the tree based on the variance captured.

Key Results and Insights

The core insight of DEITEE is the distinction between confounders and effect modifiers. While confounders are necessary to unbias training data, only effect modifiers—variables that change the magnitude of the treatment effect—are required for prediction at test time. Many variables are confounders but not effect modifiers; DEITEE identifies and discards them for the prediction phase.

Performance metrics demonstrate that DEITEE achieves low error (MSE-T) with very few features, effectively identifying effect modifiers.

Key findings from evaluations on 77 semi-synthetic datasets and real-world CDC birth data include:

  • Massive Feature Reduction: On the ACIC datasets, DEITEE reduced the required features from 58 to an average of just 3 to 5 variables (an 88-91% reduction).
  • Accuracy Preservation: The distilled models maintained accuracy levels nearly identical to the complex base models, despite the massive reduction in input data.
  • Clinical Relevance: In the birth weight study, the model identified alcohol consumption as a primary effect modifier, prioritizing it in the decision tree to optimize the information gain for ITE estimation.

Conclusion

DEITEE demonstrates that high-fidelity causal inference does not necessitate high-dimensional data at the point of care. By leveraging distillation, the method identifies the minimal set of effect modifiers needed for accurate prediction. While currently limited to binary decision trees and potentially challenged by highly smooth response surfaces, DEITEE provides a robust framework for deploying causal models in data-constrained environments.

Summary updated

Abstract

The potential for using machine learning algorithms as a tool for suggesting optimal interventions has fueled significant interest in developing methods for estimating heterogeneous or individual treatment effects (ITEs) from observational data. While several methods for estimating ITEs have been recently suggested, these methods assume no constraints on the availability of data at the time of deployment or test time. This assumption is unrealistic in settings where data acquisition is a significant part of the analysis pipeline, meaning data about a test case has to be collected in order to predict the ITE. In this work, we present Data Efficient Individual Treatment Effect Estimation (DEITEE), a method which exploits the idea that adjusting for confounding, and hence collecting information about confounders, is not necessary at test time. DEITEE allows the development of rich models that exploit all variables at train time but identifies a minimal set of variables required to estimate the ITE at test time. Using 77 semi-synthetic datasets with varying data generating processes, we show that DEITEE achieves significant reductions in the number of variables required at test time with little to no loss in accuracy. Using real data, we demonstrate the utility of our approach in helping soon-to-be mothers make planning and lifestyle decisions that will impact newborn health.
BibTeX
@inproceedings{makar2019distillation,
  author = {Maggie Makar and Adith Swaminathan and Emre Kiciman},
  title = {{A Distillation Approach to Data Efficient Individual Treatment Effect Estimation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2019},
  url = {https://openreview.net/forum?id=sQc4_hEG3Z3},
}
Figure