De-Biased Machine Learning of Global and Local Parameters Using\n Regularized Riesz Representers

Victor Chernozhukov, Whitney K. Newey, Rahul SingharXiv 2018 · DOI · arXiv

Abstract

This paper introduces a unified framework for adaptive inference on both regular (global) and non-regular (local) linear functionals of conditional expectation functions. By utilizing $\ell_1$-regularized Riesz representers within a Neyman orthogonal framework, the authors provide a method for "de-biasing" machine learning estimators. This approach ensures asymptotic uniform validity and "honest" confidence bands, even in high-dimensional settings where the underlying models are complex. The analysis yields a "double sparsity robustness" property, allowing for valid inference even when one of the nuisance components is highly complex.

Introduction

In modern data science, machine learning (ML) models are frequently deployed to predict outcomes in complex environments. However, when the goal shifts from prediction to inference—such as estimating the impact of a policy or a treatment—standard ML "plug-in" estimators often fail. These models are typically tuned to minimize mean squared error, which introduces regularization bias and overfitting that can lead to misleading conclusions.

Double Machine Learning (DML) addresses this by using "orthogonal scores" to cancel out first-order biases. A critical component of these scores is the Riesz representer (RR), a mathematical object that functions as a bias-correction term. Historically, researchers had to derive these RRs analytically for every new problem, a task that is often impossible for complex, "local" parameters (e.g., the effect of a policy on a specific income bracket). This paper automates this process, treating the RR as a nuisance parameter that can be learned directly from data.

Methodology: Automating the Riesz Representer

The core innovation of this work is the automatic estimation of the Riesz representer, extending de-biased ML to a much broader class of parameters.

1. Automatic RR Estimation

Instead of manual derivation, the authors employ a Generalized Dantzig Selector (GDS) with $\ell_1$ regularization. This allows the framework to learn the Riesz representer directly from the data's empirical moment conditions. This automation makes DML a "black-box" tool applicable to a wide range of linear functionals without requiring deep theoretical derivations for each use case.

2. Neyman Orthogonality and Cross-Fitting

The framework constructs a target equation that satisfies Neyman orthogonality, meaning the estimator is insensitive to small errors in the nuisance parameters (the regression function and the RR). To eliminate overfitting bias, the authors utilize cross-fitting: the data is split, the ML models are trained on one portion, and the parameter of interest is estimated on another.

3. Double Sparsity Robustness

A key theoretical contribution is "double sparsity robustness." The authors prove that the method remains valid as long as either the regression function or the Riesz representer is "sparse" (depends on few variables). Even if one component is "completely dense" and highly complex, the estimator can still provide valid results if the other component is sufficiently simple.

Empirical Applications

The authors demonstrate the framework's utility through two real-world economic studies, highlighting its ability to handle both population-wide and subgroup-specific effects.

401(k) Eligibility and Financial Assets

Using the 1991 US Survey of Income and Program Participation (SIPP), the authors estimated the effect of 401(k) eligibility on net financial assets. While they found a global Average Treatment Effect (ATE) of **$7,608** (SE $1,395$), their "local" analysis revealed significant heterogeneity. The effect was statistically insignificant for the second income quintile but very high for the first and fifth quintiles, demonstrating the power of the framework to uncover nuanced subgroup dynamics.

Gasoline Demand Elasticity

In an analysis of the Canadian National Private Vehicle Use Survey, the authors estimated the price elasticity of gasoline demand. While simple OLS models often produce positive elasticities—suggesting demand rises with price, which contradicts economic theory—the de-biased ML approach corrected this bias. They estimated an average price elasticity of -0.28 (SE $0.06$), providing a theoretically sound result that reaches the semi-parametric efficiency bound.

Conclusion and Future Directions

This research provides a robust framework for performing valid inference with machine learning. By automating the estimation of Riesz representers, the authors have created a unified theory that covers both global and local parameters with honest confidence bands.

While the current framework focuses on linear functionals, it sets the stage for future work in nonlinear problems and the application of non-asymptotic bounds to a wider variety of "black-box" learners, such as Random Forests or Deep Neural Networks. This work represents a significant step toward making high-dimensional statistical inference more accessible and reliable for researchers.

Summary updated

Abstract

We provide adaptive inference methods, based on $\\ell_1$ regularization, for\nregular (semi-parametric) and non-regular (nonparametric) linear functionals of\nthe conditional expectation function. Examples of regular functionals include\naverage treatment effects, policy effects, and derivatives. Examples of\nnon-regular functionals include average treatment effects, policy effects, and\nderivatives conditional on a covariate subvector fixed at a point. We construct\na Neyman orthogonal equation for the target parameter that is approximately\ninvariant to small perturbations of the nuisance parameters. To achieve this\nproperty, we include the Riesz representer for the functional as an additional\nnuisance parameter. Our analysis yields weak ``double sparsity robustness'':\neither the approximation to the regression or the approximation to the\nrepresenter can be ``completely dense'' as long as the other is sufficiently\n``sparse''. Our main results are non-asymptotic and imply asymptotic uniform\nvalidity over large classes of models, translating into honest confidence bands\nfor both global and local parameters.\n
BibTeX
@misc{chernozhukov2018de,
  author = {Victor Chernozhukov and Whitney K. Newey and Rahul Singh},
  title = {{De-Biased Machine Learning of Global and Local Parameters Using\textbackslash\{\}n Regularized Riesz Representers}},
  year = {2018},
  eprint = {1802.08667},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1802.08667},
}
Figure