High-Dimensional Econometrics and Regularized GMM

Alexandre Belloni, Victor Chernozhukov, Denis Chetverikov, Christian Hansen, Kengo Kato

Abstract

This chapter presents key concepts and theoretical results for analyzing estimation and inference in high-dimensional models. High-dimensional models are characterized by having a number of unknown parameters that is not vanishingly small relative to the sample size. We first present results in a framework where estimators of parameters of interest may be represented directly as approximate means. Within this context, we review fundamental results including high-dimensional central limit theorems, bootstrap approximation of high-dimensional limit distributions, and moderate deviation theory. We also review key concepts underlying inference when many parameters are of interest such as multiple testing with family-wise error rate or false discovery rate control. We then turn to a general high-dimensional minimum distance framework with a special focus on generalized method of moments problems where we present results for estimation and inference about model parameters. The presented results cover a wide array of econometric applications, and we discuss several leading special cases including high-dimensional linear regression and linear instrumental variables models to illustrate the general results.

Introduction: The High-Dimensional Challenge

Modern econometric analysis is increasingly characterized by "big data," where the number of potential regressors or instruments $p$ is large relative to the sample size $n$. In such high-dimensional regimes, classical statistical methods—which rely on $p$ being small and fixed—often fail to provide valid inference. This paper, a comprehensive chapter by Belloni et al., establishes a unified theoretical framework for estimation and inference in these settings, bridging the gap between machine learning regularization and structural econometric modeling.

In traditional settings, we rely on the Central Limit Theorem (CLT) to ensure that our estimators are approximately normally distributed. However, as $p$ grows, the bias introduced by regularization techniques (like Lasso) and the accumulation of estimation errors across many parameters can lead to invalid confidence intervals and misleading p-values. The authors address this by focusing on "honest" inference: methods that remain valid even when the model selection process is imperfect.

The Many Approximate Means (MAM) Framework

The first pillar of the authors' approach is the Many Approximate Means (MAM) framework. This covers estimators that can be represented as an average of influence functions plus a negligible remainder. The key innovation here is a High-Dimensional CLT that provides Gaussian approximations for the maxima of these means.

Crucially, this approximation holds even when $p$ is exponentially larger than $n$ (e.g., $\log p = o(n^{1/7})$), provided the distribution of the data has sufficiently thick tails. This is a significant departure from classical theory, which typically requires $p^2/n \to 0$. To make this operational, the authors advocate for the Gaussian multiplier bootstrap. Unlike traditional bootstraps that might require re-running expensive high-dimensional optimizations, the multiplier bootstrap allows for efficient computation of critical values for simultaneous confidence intervals and multiple testing by simulating Gaussian processes.

Regularized GMM and the De-biasing Strategy

For structural models, the authors extend the Generalized Method of Moments (GMM) to high dimensions using $\ell_1$-regularization. The primary challenge with Regularized GMM (RGMM) is the inherent bias toward zero caused by the penalty term, which is designed for prediction rather than inference.

To overcome this, the authors employ Double/De-biased RGMM. This technique utilizes Neyman Orthogonality, a property where the moment conditions are constructed to be insensitive to first-order perturbations in the high-dimensional nuisance parameters. By solving a "de-biasing" problem—effectively projecting the target score onto the orthogonal complement of the nuisance scores—the authors transform a biased regularized estimate into one that behaves like a simple approximate mean. This allows for the construction of standard errors and confidence intervals that are robust to the selection of control variables, ensuring that the machine learning component does not contaminate the structural parameters of interest.

Multiple Testing and Error Control

In high-dimensional discovery (e.g., identifying significant genes or policy impacts across many subgroups), the risk of false positives is high. The paper contrasts two main approaches:

  • Family-Wise Error Rate (FWER): Controlling the probability of making even one false discovery. While rigorous, this is often too conservative in ultra-high dimensions, potentially leading to zero results.
  • False Discovery Rate (FDR): Controlling the expected proportion of false discoveries among all rejected hypotheses.

The authors provide robust procedures for FDR control that remain valid under the dependencies typical of econometric models, ensuring that "discoveries" are statistically meaningful even when testing thousands of hypotheses simultaneously.

Applications and Empirical Relevance

The theoretical results are illustrated through several key econometric models and datasets:

  1. High-Dimensional Linear Regression: Selecting from a vast pool of potential control variables to estimate a treatment effect without suffering from omitted variable bias or over-fitting.
  2. Instrumental Variables (IV): Handling many instruments or many endogenous variables, a common problem in empirical macroeconomics and labor economics.
  3. Policy Analysis (Tennessee STAR Project): The paper cites this as a motivating example for analyzing many groups and student-teacher pairs where high-dimensional controls are necessary.
  4. Genoeconomics: Managing millions of SNPs to find genetic correlates of economic behavior, where FDR control is paramount.

Conclusion and Future Directions

The framework presented by Belloni et al. provides the necessary rigor for applying machine learning tools to causal and structural questions in economics. By emphasizing de-biasing and high-dimensional CLTs, the authors ensure that researchers can leverage the predictive power of high-dimensional models without sacrificing the validity of their statistical conclusions.

However, limitations remain. Most results rely on approximate sparsity—the assumption that only a small subset of variables is truly influential. If the true model is "dense," these methods may lose efficiency. Additionally, the computational cost of de-biasing every parameter in massive systems and extending these results to highly non-convex loss landscapes remain active areas for future research.

Summary updated

Abstract

This chapter presents key concepts and theoretical results for analyzing estimation and inference in high-dimensional models. High-dimensional models are characterized by having a number of unknown parameters that is not vanishingly small relative to the sample size. We first present results in a framework where estimators of parameters of interest may be represented directly as approximate means. Within this context, we review fundamental results including high-dimensional central limit theorems, bootstrap approximation of high-dimensional limit distributions, and moderate deviation theory. We also review key concepts underlying inference when many parameters are of interest such as multiple testing with family-wise error rate or false discovery rate control. We then turn to a general high-dimensional minimum distance framework with a special focus on generalized method of moments problems where we present results for estimation and inference about model parameters. The presented results cover a wide array of econometric applications, and we discuss several leading special cases including high-dimensional linear regression and linear instrumental variables models to illustrate the general results.
BibTeX
@misc{belloni2018high,
  author = {Alexandre Belloni and Victor Chernozhukov and Denis Chetverikov and Christian Hansen and Kengo Kato},
  title = {{High-Dimensional Econometrics and Regularized GMM}},
  year = {2018},
  eprint = {1806.01888},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1806.01888},
}
Figure