Post-selection and post-regularization inference in linear models with many controls and instruments

Martin Spindler, Victor Chernozhukov, Christian Hansen

TL;DR

This paper introduces a novel statistical approach for estimating causal effects in complex linear models with a vast number of potential control and instrumental variables. Their method, dubbed "double-selection," uses modern data-mining techniques like LASSO to intelligently select relevant variables, ensuring reliable inference even when many irrelevant variables are present. This is crucial for uncovering true causal relationships in high-dimensional datasets common in economics and other fields.

What Problem Does This Paper Address?

In many scientific fields, particularly economics, researchers aim to estimate the causal effect of one variable (say, a policy or treatment) on another. A common tool for this is instrumental variables (IV) regression, which uses a third variable (the instrument) to isolate the variation in the variable of interest that is truly exogenous. However, real-world data often comes with a multitude of potential control variables (confounders) and instrumental variables, far exceeding the number of observations. This "high-dimensional" setting poses a significant challenge: how do you select the right controls and instruments from a vast pool without introducing bias or losing statistical power? Traditional methods struggle, and naive variable selection can lead to overfitting, spurious correlations, and incorrect conclusions about causal effects. This paper tackles this problem by developing a principled way to perform variable selection and then conduct reliable inference in these high-dimensional IV models.

Methods

The core of the paper's contribution lies in its "double-selection" strategy, which extends existing work on sparse models. Instead of relying on a single selection step, they propose a two-stage process:

  1. Select Controls and Instruments: They first use LASSO (Least Absolute Shrinkage and Selection Operator) or a similar post-LASSO procedure to identify relevant exogenous control variables ($x_i$) and relevant instruments ($z_i$). LASSO is a popular regularization technique that shrinks the coefficients of less important variables towards zero, effectively performing variable selection.
  2. Estimate Causal Effect: After selecting a parsimonious set of controls and instruments, they then use these selected variables in a standard IV regression framework to estimate the causal effect of interest.

Crucially, they also propose using "orthogonal moment functions." These are specially constructed estimating equations that are designed to be robust to small errors in the variable selection process. This means that even if the selection procedure makes a minor mistake (e.g., omits a variable with a very small coefficient), the final inference about the causal effect remains valid. This is a key innovation that distinguishes their approach from naive post-selection methods.

Datasets

The paper uses a simulated dataset for illustration and an empirical dataset from Berry et al. (1995) concerning automobile prices. The empirical example is not a newly introduced benchmark but rather an application of their method to an existing dataset to demonstrate its practical utility.

Key Insights and Ideas

  • Double-Selection is Key: Simply selecting variables once is insufficient. A two-stage selection process for both controls and instruments is necessary for robust inference in high-dimensional IV settings.
  • Orthogonality Guards Against Selection Errors: The use of "orthogonal moment functions" is a critical theoretical innovation. It provides a formal guarantee that inference remains valid even if the variable selection process isn't perfect, which is almost always the case in practice.
  • Modern Data Mining for Reliable Inference: The paper advocates for using principled data-mining techniques like LASSO, distinguishing them from "unprincipled data-snooping." These methods aim to find true predictive power while guarding against false discoveries and overfitting.
  • High-Dimensionality is Pervasive: The authors highlight that the problem of having more potential variables than observations is increasingly common, making their approach broadly applicable.

Key Results

The paper presents theoretical results showing that their proposed estimator is asymptotically normal and that confidence intervals based on their method are uniformly valid across a wide range of models.

In their simulation study, the "Double-Selection" estimator performed remarkably well, achieving a median bias of 0.021 and a median absolute deviation (MAD) of 0.099, closely matching the performance of an infeasible "Oracle" estimator (which knows the true model). In contrast, naive selection methods (Naive 1 and Naive 2) exhibited significantly higher bias and larger deviations.

The empirical example, applied to automobile demand, showed that their method produced more plausible estimates of price elasticity compared to standard methods, suggesting that the inclusion of more controls and instruments, selected intelligently, can lead to more economically sensible results.

Limitations and Future Work

The authors acknowledge that their theoretical results are based on an "approximate sparsity" assumption, meaning that the true signal in the data can be captured by a relatively small number of variables. While they argue this is often a reasonable assumption, it might not hold in all scenarios.

Future work could involve extending these methods to more complex models, such as those with random coefficients or non-linear relationships, as hinted at in the empirical example. Further exploration of different variable selection techniques and their impact on inference would also be valuable. The paper also suggests that their approach can be adapted to other settings beyond IV models.

Summary updated

Abstract

In this note, we offer an approach to estimating structural parameters in the presence of many instruments and controls based on methods for estimating sparse high-dimensional models. We use these high-dimensional methods to select both which instruments and which control variables to use. The approach we take extends Belloni et al. (2012), which covers selection of instruments for IV models with a small number of controls, and extends Belloni, Chernozhukov and Hansen (2014), which covers selection of controls in models where the variable of interest is exogenous conditional on observables, to accommodate both a large number of controls and a large number of instruments. We illustrate the approach with a simulation and an empirical example. Technical supporting material is available in a supplementary appendix here.
BibTeX
@inproceedings{spindler2015post,
  author = {Martin Spindler and Victor Chernozhukov and Christian Hansen},
  title = {{Post-selection and post-regularization inference in linear models with many controls and instruments}},
  booktitle = {American},
  year = {2015},
}
Figure