Inference for High-Dimensional Sparse Econometric Models

Alexandre Belloni, Victor Chernozhukov, Christian HansenarXiv 2011 · DOI · arXiv

Abstract

This article explores estimation and inference methods for high-dimensional sparse (HDS) regression models in econometrics. HDS models arise when many potential regressors are available, but the underlying regression function is well-approximated by a small, unknown subset of them. We discuss methods for identifying these regressors and estimating their coefficients using $\ell_1$-penalization (Lasso). To capture realistic practical situations, we allow for imperfect selection and study its impact on inference. We focus on instrumental variables and partially linear models, illustrating our findings with applications to returns to schooling and economic growth.

Introduction: The Challenge of High-Dimensional Data

In classical econometrics, researchers typically select variables based on established theory. However, modern "big data" contexts often present hundreds of potential regressors (e.g., complex demographic interactions or time-series lags) with relatively few observations ($p > n$). Standard regression techniques fail in these high-dimensional settings.

While machine learning tools like Lasso excel at variable selection for prediction, they often introduce "shrinkage bias," which can distort the estimated effect of specific policy variables. Furthermore, if Lasso inadvertently excludes a variable weakly correlated with the treatment, it can lead to significant omitted variable bias. This paper provides a framework for using these tools to obtain "honest" p-values and confidence intervals.

Methodology: Post-Lasso and Double Selection

The authors focus on High-Dimensional Sparse (HDS) models, assuming the true relationship depends on a small subset of available variables. To address the limitations of standard Lasso, two primary techniques are proposed:

  1. Post-Lasso Estimation: To mitigate shrinkage bias, the authors advocate for "Post-Lasso"—using Lasso solely for variable selection and then performing a standard OLS regression on the selected subset. As shown in the error analysis below, Post-Lasso significantly reduces approximation errors compared to conventional methods.
Table 1: Comparison of $L_2$ and $L_\infty$ errors for different sparse approximation methods.
  1. Double-Selection (Double-Post-Lasso): This is the paper's most influential contribution. Instead of selecting variables based only on the outcome, researchers run Lasso twice: once to select variables predicting the outcome and once to select variables predicting the treatment variable. The final regression uses the union of these sets, providing a "safety net" against omitted variable bias.

Empirical Applications

The methods are validated using classic econometric datasets, demonstrating their ability to approximate complex functions with parsimonious models.

Returns to Schooling (2000 U.S. Census)

Using 2000 U.S. Census data, the authors modeled the "earnings function" (log-wage vs. education). The sparse Post-Lasso model approximates the complex, non-linear relationship far more effectively than traditional low-order polynomials, even when restricted to the same number of coefficients.

Figure 1: Traditional vs. Lasso-based approximations of the expected wage function using 5 coefficients.

Instrumental Variables and Economic Growth

  • Angrist and Krueger (1991): In a dataset with 1,530 potential instruments, Lasso identified that only a tiny fraction were necessary. These sparse estimates were more precise than traditional methods, which often suffer from overfitting in the first stage.
  • Barro and Lee (1994): In growth data ($n=90, p=62$), the Double-Selection method confirmed the "conditional convergence" hypothesis with significantly tighter confidence intervals than previous ad-hoc selection methods.

Key Insights and Limitations

  • Sparsity as a Bridge: We don't need to know which variables matter beforehand; we only need to assume that not many of them do. This allows for robust inference in $p \gg n$ scenarios.
  • Pivotal Penalties: The authors propose "Square-root Lasso" and "Iterated Lasso," where the penalty level ($\lambda$) is automatically calibrated to data noise, simplifying practical application.

Limitations: The framework relies heavily on the sparsity assumption. If the underlying model is "dense" (where every variable has a tiny effect), these methods may underperform. Additionally, while robust, the methods still face challenges with "very weak" signals and non-IID data, such as complex time-series correlations.

Summary updated

Abstract

This article is about estimation and inference methods for high dimensional sparse (HDS) regression models in econometrics. High dimensional sparse models arise in situations where many regressors (or series terms) are available and the regression function is well-approximated by a parsimonious, yet unknown set of regressors. The latter condition makes it possible to estimate the entire regression function effectively by searching for approximately the right set of regressors. We discuss methods for identifying this set of regressors and estimating their coefficients based on $\ell_1$-penalization and describe key theoretical results. In order to capture realistic practical situations, we expressly allow for imperfect selection of regressors and study the impact of this imperfect selection on estimation and inference results. We focus the main part of the article on the use of HDS models and methods in the instrumental variables model and the partially linear model. We present a set of novel inference results for these models and illustrate their use with applications to returns to schooling and growth regression.
BibTeX
@misc{belloni2011inference,
  author = {Alexandre Belloni and Victor Chernozhukov and Christian Hansen},
  title = {{Inference for High-Dimensional Sparse Econometric Models}},
  year = {2011},
  eprint = {1201.0220},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1201.0220},
}
Figure