Abstract
We give debiased machine learners of parameters of interest that depend on generalized linear regressions, which regressions make a residual orthogonal to regressors. The parameters of interest include many causal and policy effects. We give neural net learners of the bias correction that are automatic in only depending on the object of interest and the regression residual. Convergence rates are given for these neural nets and for more general learners of the bias correction. We also give conditions for asymptotic normality and consistent asymptotic variance estimation of the learner of the object of interest.
Introduction
In modern econometrics and statistics, Machine Learning (ML) has become a powerful tool for estimating complex nuisance parameters. However, while ML models excel at prediction, they are often biased for statistical inference due to regularization and model selection. This bias can lead to unreliable confidence intervals for parameters of interest, such as causal or policy effects.
To address this, researchers have developed "Double Machine Learning" (DML) frameworks that utilize Neyman Orthogonal moment functions. These functions incorporate a "bias correction" term—mathematically known as the Riesz Representer—to ensure that the final estimate is robust to errors in the initial ML regressions.
The Challenge of Manual Derivation
A significant hurdle in applying DML is that the Riesz Representer ($\alpha_0$) typically requires a specific, hand-derived formula for every unique causal effect. For complex or non-linear models, deriving these formulas is mathematically intensive and prone to error.
As shown in Table 1, different effects like the Weighted Average Derivative or the Average Treatment Effect (ATE) require distinct functional forms for their components. The goal of this research is to automate the discovery of these correction terms, removing the need for manual derivation.
Automatic Debiased Machine Learning (Auto-DML)
The authors propose Automatic Debiased Machine Learning (Auto-DML), a framework that leverages neural networks to learn the bias-correction term directly from the data. Instead of deriving $\alpha_0$ analytically, the method identifies an objective function that $\alpha_0$ naturally minimizes.
Key features of the Auto-DML approach include:
- Neural Net Learners: Multi-Layer Perceptrons (MLPs) with ReLU activations are used to minimize the objective function and "learn" the Riesz Representer.
- Generalized Linear Regression: The framework is broad enough to cover any regression where the residual is orthogonal to the regressors, including standard GLMs and quantile regressions.
- Cross-Fitting: To prevent overfitting, the data is split into groups. Models are trained on one subset and evaluated on another, ensuring that nuisance estimates do not memorize noise from the inference sample.
Theoretical Guarantees and Results
The paper provides rigorous mathematical foundations for the Auto-DML estimator. Key results include:
- Convergence Rates: The authors derive mean-square convergence rates for the neural net learners. They demonstrate that if the underlying functions are sufficiently "smooth," the neural net can learn the correction term fast enough to support valid inference.
- Asymptotic Normality: The Auto-DML estimator is proven to be "root-n consistent" and follows a normal distribution in large samples. This allows researchers to construct standard 95% confidence intervals for policy effects.
- Consistent Variance Estimation: The framework includes a method for consistent asymptotic variance estimation, which is essential for calculating the precision of the statistical estimates.
Insights and Limitations
The "automatic" nature of this framework is its most significant contribution. By showing that the bias correction depends only on the object of interest and the regression residual, the authors provide a "plug-and-play" solution for high-dimensional inference. Furthermore, the orthogonal moment functions are "model-free," making the resulting standard errors robust to misspecification.
However, some limitations remain:
- Smoothness Requirements: The theoretical guarantees rely on the underlying data-generating processes being sufficiently smooth. Highly discontinuous data may hinder convergence.
- Computational Cost: Training deep neural networks across multiple folds of a cross-fitting procedure is computationally more demanding than simpler methods like Lasso.
- Future Directions: While this work focuses on neural networks, the authors suggest that the same "automatic" objective function could potentially be applied to other learners like Random Forests or Boosting.
Conclusion
Auto-DML represents a significant step forward in making advanced causal inference accessible. By automating the debiasing process through neural networks, it bridges the gap between the predictive power of deep learning and the rigorous requirements of statistical inference.
Summary updated