Uniform post selection inference for LAD regression and other Z-estimation problems

Kengo Kato, Victor Chernozhukov, Alexandre Belloni

TL;DR

This paper introduces a new statistical method for creating reliable confidence intervals in high-dimensional regression problems, especially when dealing with median regression (Least Absolute Deviation). The key innovation is an "orthogonalization" technique that makes the inference robust even when the initial model selection step isn't perfect, a common issue in complex, high-dimensional settings. This leads to uniformly valid and semiparametrically efficient confidence regions, a significant improvement over existing methods that often fail in such scenarios.

What Problem Does This Paper Address?

In modern data analysis, we often encounter situations with a vast number of potential predictor variables (high-dimensional data) and we're interested in understanding the relationship between a few specific variables and an outcome. A common goal is to estimate the effect of a "treatment" or key regressor (like in program evaluation) on an outcome, while controlling for many other factors. Median regression (Least Absolute Deviation or LAD) is a robust alternative to standard linear regression, particularly when errors are not normally distributed.

However, when dealing with high-dimensional data, we often first perform "model selection" to choose which of the many potential predictors to include in the final model. This selection process, especially when using methods like Lasso, can introduce biases and lead to incorrect confidence intervals if not handled carefully. Traditional inference methods break down because they assume the selected model is the "true" model, which is rarely the case. The paper addresses the critical problem of developing statistical inference (specifically, confidence regions) that remains valid and reliable even after a model selection step has been performed, and does so uniformly across a wide range of possible data-generating processes, including those where the "true" model is not perfectly identifiable.

Methods

The core of the paper's contribution lies in a novel approach to constructing confidence regions for regression coefficients. The authors leverage Neyman's orthogonalization to create a moment equation that is "immunized" against estimation errors in the nuisance parameters (the parts of the model we're not directly interested in). This is achieved by ensuring that the score function (related to the derivative of the likelihood or moment condition) is orthogonal to small perturbations in the nuisance parameters.

Specifically, the method involves three main steps:

  1. Estimating the confounding function: This involves estimating the relationship between the treatment variable and other controls, often using penalized regression techniques like Lasso or post-Lasso.
  2. Estimating instruments: This step generates "instruments" that are residuals from a regression of the treatment variable on the controls.
  3. Instrumental median regression: The final step uses these instruments in a median regression framework to estimate the target coefficient. This instrumental approach, combined with the orthogonalization, ensures that even if the initial model selection (Step 1) is imperfect (e.g., misses some relevant variables or includes irrelevant ones), the final inference remains valid.

The paper also generalizes this approach to a broader class of "Z-estimation problems" where there can be multiple target parameters (p1) that are potentially much larger than the sample size (n), and extends the theory to "approximately sparse" models, where coefficients are small but not exactly zero.

Datasets

Not discussed in the paper. The paper focuses on theoretical development and uses simulated data for Monte Carlo experiments.

Key Insights and Ideas

  • Robustness through Orthogonalization: The key insight is that by constructing an "orthogonalized" moment condition, the inference procedure becomes robust to imperfect model selection. This means we don't need perfect variable selection for reliable confidence intervals.
  • Uniform Validity: The proposed method provides confidence regions that are valid uniformly across a broad class of data-generating processes, not just for a specific, pre-selected model. This is a significant theoretical achievement.
  • Semiparametric Efficiency: The method achieves semiparametric efficiency, meaning it extracts as much information as possible from the data for estimating the target parameter, given the model structure.
  • Handling High Dimensions and Many Parameters: The framework is designed to work even when the number of potential predictors (p) or the number of target parameters (p1) is much larger than the sample size (n), a common scenario in modern data.
  • Approximate Sparsity: The theory extends beyond strictly "sparse" models (where many coefficients are exactly zero) to "approximately sparse" models, where coefficients are small but non-zero. This is more realistic in many applications.

Key Results

  • The proposed instrumental median regression estimator is shown to be asymptotically normally distributed uniformly with respect to the underlying sparse model.
  • The method achieves semiparametric efficiency.
  • The paper establishes uniform asymptotic normality for estimators in a general Z-estimation framework with potentially many target parameters (p1 >> n).
  • Simultaneous confidence bands are constructed for multiple target parameters, with guaranteed asymptotic validity.
  • Monte Carlo simulations demonstrate that the proposed method achieves accurate coverage rates (close to nominal levels) even in scenarios where standard post-selection inference procedures fail dramatically due to imperfect model selection. The proposed estimator also exhibits lower bias and standard deviation.

Limitations and Future Work

  • Homoscedasticity Assumption: The primary theoretical results are derived under the assumption of homoscedastic errors (constant variance). The paper mentions that generalization to heteroscedastic errors is possible but requires more complex techniques (weighted versions of auxiliary equations) and is developed in a separate working paper.
  • Sparsity Requirements: While the theory extends to approximately sparse models, it still relies on some form of sparsity (either in the true parameters or in the estimators). The exact conditions under which the method performs best are characterized by specific growth rates of dimensions (p, p1, s) relative to the sample size (n).
  • Computational Complexity: While the algorithms are described as computationally tractable (involving convex optimization and one-dimensional search), the practical implementation for very high dimensions might still be computationally intensive.
  • Future Work: The authors suggest extensions to heteroscedastic cases and further exploration of the theory for approximate sparsity. They also mention the development of alternative implementations and the application to a broader range of semi-parametric models.

Summary updated

Abstract

We develop uniformly valid confidence regions for regression coefficients in a highdimensional sparse median regression model with homoscedastic errors. Our methods are based on a moment equation that is immunized against non-regular estimation of the nuisance part of the median regression function by using Neyman's orthogonalization. We establish that the resulting instrumental median regression estimator of a target regression coefficient is asymptotically normally distributed uniformly with respect to the underlying sparse model and is semiparametrically efficient. We also generalize our method to a general non-smooth Z-estimation framework with the number of target parameters p1 being possibly much larger than the sample size n. We extend Huber's results on asymptotic normality to this setting, demonstrating uniform asymptotic normality of the proposed estimators over p1-dimensional rectangles, constructing simultaneous confidence bands on all of the p1 target parameters, and establishing asymptotic validity of the bands uniformly over underlying approximately sparse models.
BibTeX
@misc{kato2017uniform,
  author = {Kengo Kato and Victor Chernozhukov and Alexandre Belloni},
  title = {{Uniform post selection inference for LAD regression and other Z-estimation problems}},
  year = {2017},
  doi = {10.1920/wp.cem.2014.5114},
}
Figure