Automatic Debiased Machine Learning via Riesz Regression

Victor Chernozhukov, Whitney K. Newey, Victor Quintas-Martinez, Vasilis SyrgkanisarXiv 2021 · DOI · arXiv

Abstract

We give debiased machine learners of parameters of interest that depend on generalized linear regressions, which regressions make a residual orthogonal to regressors. The parameters of interest include many causal and policy effects. We give neural net learners of the bias correction that are automatic in only depending on the object of interest and the regression residual. Convergence rates are given for these neural nets and for more general learners of the bias correction. We also give conditions for asymptotic normality and consistent asymptotic variance estimation of the learner of the object of interest.

Introduction

In modern econometrics and statistics, Machine Learning (ML) has become a powerful tool for estimating complex nuisance parameters. However, while ML models excel at prediction, they are often biased for statistical inference due to regularization and model selection. This bias can lead to unreliable confidence intervals for parameters of interest, such as causal or policy effects.

To address this, researchers have developed "Double Machine Learning" (DML) frameworks that utilize Neyman Orthogonal moment functions. These functions incorporate a "bias correction" term—mathematically known as the Riesz Representer—to ensure that the final estimate is robust to errors in the initial ML regressions.

The Challenge of Manual Derivation

A significant hurdle in applying DML is that the Riesz Representer ($\alpha_0$) typically requires a specific, hand-derived formula for every unique causal effect. For complex or non-linear models, deriving these formulas is mathematically intensive and prone to error.

Table 1: Examples of effects and their corresponding components in the Auto-DML framework.

As shown in Table 1, different effects like the Weighted Average Derivative or the Average Treatment Effect (ATE) require distinct functional forms for their components. The goal of this research is to automate the discovery of these correction terms, removing the need for manual derivation.

Automatic Debiased Machine Learning (Auto-DML)

The authors propose Automatic Debiased Machine Learning (Auto-DML), a framework that leverages neural networks to learn the bias-correction term directly from the data. Instead of deriving $\alpha_0$ analytically, the method identifies an objective function that $\alpha_0$ naturally minimizes.

Key features of the Auto-DML approach include:

  • Neural Net Learners: Multi-Layer Perceptrons (MLPs) with ReLU activations are used to minimize the objective function and "learn" the Riesz Representer.
  • Generalized Linear Regression: The framework is broad enough to cover any regression where the residual is orthogonal to the regressors, including standard GLMs and quantile regressions.
  • Cross-Fitting: To prevent overfitting, the data is split into groups. Models are trained on one subset and evaluated on another, ensuring that nuisance estimates do not memorize noise from the inference sample.

Theoretical Guarantees and Results

The paper provides rigorous mathematical foundations for the Auto-DML estimator. Key results include:

  1. Convergence Rates: The authors derive mean-square convergence rates for the neural net learners. They demonstrate that if the underlying functions are sufficiently "smooth," the neural net can learn the correction term fast enough to support valid inference.
  2. Asymptotic Normality: The Auto-DML estimator is proven to be "root-n consistent" and follows a normal distribution in large samples. This allows researchers to construct standard 95% confidence intervals for policy effects.
  3. Consistent Variance Estimation: The framework includes a method for consistent asymptotic variance estimation, which is essential for calculating the precision of the statistical estimates.

Insights and Limitations

The "automatic" nature of this framework is its most significant contribution. By showing that the bias correction depends only on the object of interest and the regression residual, the authors provide a "plug-and-play" solution for high-dimensional inference. Furthermore, the orthogonal moment functions are "model-free," making the resulting standard errors robust to misspecification.

However, some limitations remain:

  • Smoothness Requirements: The theoretical guarantees rely on the underlying data-generating processes being sufficiently smooth. Highly discontinuous data may hinder convergence.
  • Computational Cost: Training deep neural networks across multiple folds of a cross-fitting procedure is computationally more demanding than simpler methods like Lasso.
  • Future Directions: While this work focuses on neural networks, the authors suggest that the same "automatic" objective function could potentially be applied to other learners like Random Forests or Boosting.

Conclusion

Auto-DML represents a significant step forward in making advanced causal inference accessible. By automating the debiasing process through neural networks, it bridges the gap between the predictive power of deep learning and the rigorous requirements of statistical inference.

Summary updated

Abstract

A variety of interesting parameters may depend on high dimensional regressions. Machine learning can be used to estimate such parameters. However estimators based on machine learners can be severely biased by regularization and/or model selection. Debiased machine learning uses Neyman orthogonal estimating equations to reduce such biases. Debiased machine learning generally requires estimation of unknown Riesz representers. A primary innovation of this paper is to provide Riesz regression estimators of Riesz representers that depend on the parameter of interest, rather than explicit formulae, and that can employ any machine learner, including neural nets and random forests. End-to-end algorithms emerge where the researcher chooses the parameter of interest and the machine learner and the debiasing follows automatically. Another innovation here is debiased machine learners of parameters depending on generalized regressions, including high-dimensional generalized linear models. An empirical example of automatic debiased machine learning using neural nets is given. We find in Monte Carlo examples that automatic debiasing sometimes performs better than debiasing via inverse propensity scores and never worse. Finite sample mean square error bounds for Riesz regression estimators and asymptotic theory are also given.
BibTeX
@misc{chernozhukov2021automatic,
  author = {Victor Chernozhukov and Whitney K. Newey and Victor Quintas-Martinez and Vasilis Syrgkanis},
  title = {{Automatic Debiased Machine Learning via Riesz Regression}},
  year = {2021},
  eprint = {2104.14737},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2104.14737},
}
Figure