High Dimensional Sparse Econometric Models: An Introduction

Alexandre Belloni, Victor ChernozhukovarXiv 2011 · DOI · arXiv

Abstract: In this chapter we discuss conceptually high dimensional sparse econometric models as well as estimation of these models using L1-penalization and post-L1-penalization methods. Focusing on linear and nonparametric regression frameworks, we discuss various econometric examples, present basic theoretical results, and illustrate the concepts and methods with Monte Carlo simulations and an empirical application. In the application, we examine and confirm the empirical validity of the Solow-Swan model for international economic growth.

Introduction: Addressing the "Big Data" Challenge in Econometrics

In traditional econometrics, researchers typically work with a small, hand-picked set of variables. However, modern datasets often present a "Big Data" challenge: we might have hundreds of potential predictors (regressors) but only a few dozen observations. When the number of variables ($p$) is larger than the sample size ($n$), standard methods like Ordinary Least Squares (OLS) fail completely.

Historically, researchers handled this by using "ad hoc" selection—picking variables based on intuition or simple correlations. This leads to "data snooping" biases and results that are difficult to replicate. Belloni and Chernozhukov address this by formalizing the High Dimensional Sparse (HDS) framework. They assume that while there are many potential variables, only a small subset ($s$) are actually necessary to capture the essential features of the data. The challenge is finding that "needle in the haystack" in a mathematically rigorous way.

Methodology: LASSO and Post-LASSO

The paper focuses on two primary estimation techniques designed to handle high-dimensional sparsity:

  1. LASSO ($\ell_1$-penalization): This method adds a penalty term to the least squares objective function proportional to the sum of the absolute values of the coefficients. This "shrinks" less important coefficients to exactly zero, effectively performing variable selection and estimation simultaneously.
  2. Post-LASSO: While LASSO is excellent for selection, the penalty term introduces a "shrinkage bias," pulling the remaining coefficients too close to zero. Post-LASSO solves this by using LASSO only for variable selection, then running a standard OLS regression on the selected subset to produce unbiased estimates.

Additionally, the authors introduce an iterative algorithm for estimating the noise level ($\sigma$). Since the optimal penalty level for LASSO depends on the unknown noise in the data, this data-driven approach makes the methods practical for real-world empirical work.

Key Insights: Approximate Sparsity and Selection Geometry

A central argument of the HDS framework is approximate sparsity. Even if a model isn't "perfectly" sparse (i.e., some small effects exist for every variable), a sparse model often provides a much better approximation than a conventional low-dimensional model.

The paper explains the geometry of selection using the $\ell_1$ penalty, which produces sparsity because its "corners" are more likely to hit the OLS error ellipse than the smooth curves of an $\ell_2$ (Ridge) penalty. Furthermore, Post-LASSO often performs better than LASSO in practice because it discards the shrinkage bias while retaining the benefits of a parsimonious model.

Empirical Applications and Results

The authors validate their approach using several classic econometric contexts, including 2000 U.S. Census data for earnings regressions, Angrist and Krueger's (1991) return to schooling data, and Barro and Lee's (1994) cross-country growth dataset.

Earnings Function Approximation

In the U.S. Census example, the authors demonstrate how sparse models outperform traditional low-order polynomials in approximating complex wage functions.

Error metrics for conventional vs. sparse approximations

As shown in the table above, a Post-LASSO model with just 5 terms achieved an $L_\infty$ error of 0.0788, compared to 0.2896 for a conventional 5-term polynomial—a massive improvement in accuracy for the same model complexity.

Traditional vs LASSO approximations with 5 coefficients

The visual comparison illustrates how the Post-LASSO approximation tracks the expected wage function much more closely than the traditional approach, especially in capturing the non-linearities present at higher education levels.

Growth Convergence and Simulations

In the empirical application to international growth, the authors confirmed the "conditional convergence" hypothesis. After using LASSO to select from 62 variables, they found a statistically significant negative coefficient for initial GDP, supporting the theory that poorer countries catch up to richer ones when other factors are held constant. In Monte Carlo tests, Post-LASSO reduced the mean bias by roughly 75% to 95% compared to standard LASSO, depending on the noise level in the design.

Limitations and Future Directions

Despite its power, the HDS framework has limitations. The theoretical guarantees rely on the "Restricted Sparse Eigenvalue" condition, which requires that potential predictors are not too highly correlated. If variables are nearly identical, LASSO may struggle to select the correct one. Furthermore, performing standard hypothesis testing (like calculating p-values) after variable selection is a complex, ongoing area of research. Finally, these methods assume the underlying truth is sparse; if the true model involves hundreds of tiny, equally important effects, sparse methods will inherently miss the full picture.

Summary updated

Abstract

In this chapter we discuss conceptually high dimensional sparse econometric models as well as estimation of these models using L1-penalization and post-L1-penalization methods. Focusing on linear and nonparametric regression frameworks, we discuss various econometric examples, present basic theoretical results, and illustrate the concepts and methods with Monte Carlo simulations and an empirical application. In the application, we examine and confirm the empirical validity of the Solow-Swan model for international economic growth.
BibTeX
@misc{belloni2011high,
  author = {Alexandre Belloni and Victor Chernozhukov},
  title = {{High Dimensional Sparse Econometric Models: An Introduction}},
  year = {2011},
  eprint = {1106.5242},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1106.5242},
}
Figure