Uniform post-selection inference for least absolute deviation regression and other Z-estimation problems

Alexandre Belloni, Victor Chernozhukov, Kengo Kato

Introduction

In modern data science, researchers frequently encounter "high-dimensional" settings where the number of potential predictors ($p$) significantly exceeds the number of observations ($n$). While sparsity-inducing methods like the Lasso have become standard for variable selection, performing valid statistical inference—such as constructing reliable confidence intervals for specific coefficients—remains a significant challenge.

The primary obstacle is "post-selection" bias. Traditional inference often assumes an "oracle" property, where the selection step is presumed to have perfectly identified the true underlying model. In practice, however, selection algorithms often miss "moderate" variables—those with effects too small to be detected by the Lasso but large enough to bias the estimates of other parameters if omitted. This leads to "fragile" inference, characterized by overly narrow confidence intervals and inflated Type I error rates.

The Robustness of Median Regression

This paper focuses on Least Absolute Deviation (LAD) regression, also known as median regression. LAD is highly valued for its robustness to outliers in the dependent variable, making it a preferred alternative to Ordinary Least Squares (OLS) in many empirical settings. However, LAD presents unique mathematical difficulties in high dimensions because its objective function is non-smooth (it involves absolute values rather than squares), making standard Taylor expansion techniques inapplicable.

Methodology: Immunized Estimation via Neyman’s Orthogonalization

To address these challenges, the authors propose a three-step "immunized" estimation procedure designed to provide valid inference even when variable selection is imperfect. The core of the approach is Neyman’s orthogonalization, which constructs a score function that is insensitive to small errors in the estimation of nuisance parameters.

The procedure follows three main steps:

  1. Nuisance Estimation: The relationship between the outcome and the "control" variables is estimated using $\ell_1$-penalized median regression.
  2. Instrument Construction: An "instrument" for the target variable of interest is created by regressing it on the other control variables using Lasso or Post-Lasso techniques.
  3. Orthogonal Estimation: These components are combined into an "orthogonal moment equation."

By "immunizing" the moment equation, the resulting estimator for the target coefficient remains asymptotically normal and unbiased. This allows the researcher to tolerate the inevitable mistakes that occur during the selection process when $p \gg n$.

Theoretical Generalization: Z-Estimation

Beyond median regression, the authors extend their framework to a general class of non-smooth Z-estimation problems. This includes estimators defined by setting a system of equations to zero, such as Huber’s M-estimators. The paper demonstrates that this method can handle a large number of target parameters ($p_1 \gg n$), providing simultaneous confidence bands that are valid uniformly over a wide range of sparse models. This "uniformity" is crucial; it ensures the method remains reliable even when coefficients are "local to zero," a scenario where traditional post-selection methods typically fail.

Key Results and Performance

The effectiveness of the proposed method was evaluated through extensive Monte Carlo experiments. The simulations utilized designs with 300 control variables and only 250 observations, testing the method across 100 different data-generating processes with varying correlations and signal strengths.

The results highlight several key advantages:

  • Reliable Rejection Rates: While standard post-selection methods exhibited false rejection rates as high as 50% for true null hypotheses, the immunized estimator consistently maintained the nominal 5% level.
  • Reduced Bias: The proposed estimator showed significantly lower mean bias and root mean square error (RMSE) compared to naive alternatives, particularly in cases where the regressor of interest was highly correlated with other controls.
  • Efficiency: Despite the added robustness, the estimator is semiparametrically efficient, meaning it achieves the best possible precision for this class of problems without sacrificing power.

Limitations and Future Directions

While the method represents a significant advancement in high-dimensional inference, it is not without limitations. It fundamentally relies on the sparsity assumption—the idea that the true model can be approximated by a relatively small number of variables. Additionally, the current results primarily assume homoscedastic errors. While the authors outline a path for heteroscedastic cases, they note that estimating the required weights is a delicate process that remains an area for future research. Finally, the three-step procedure is computationally more intensive than simple penalized regression, though it remains tractable for modern computing environments.

Conclusion

This research provides a rigorous framework for performing reliable inference in complex, high-dimensional settings. By leveraging Neyman’s orthogonalization, the authors bridge the gap between variable selection and valid statistical testing, offering a robust tool for researchers working with non-smooth objective functions and large-scale datasets.

Summary updated

Abstract

We develop uniformly valid confidence regions for regression coefficients in a high-dimensional sparse median regression model with homoscedastic errors. Our methods are based on a moment equation that is immunized against nonregular estimation of the nuisance part of the median regression function by using Neyman’s orthogonalization. We establish that the resulting instrumental median regression estimator of a target regression coefficient is asymptotically normally distributed uniformly with respect to the underlying sparse model and is semiparametrically efficient. We also generalize our method to a general nonsmooth Z-estimation framework where the number of target parameters is possibly much larger than the sample size. We extend Huber's results on asymptotic normality to this setting, demonstrating uniform asymptotic normality of the proposed estimators over rectangles, constructing simultaneous confidence bands on all of the target parameters, and establishing asymptotic validity of the bands uniformly over underlying approximately sparse models.
BibTeX
@inproceedings{belloni2014uniform,
  author = {Alexandre Belloni and Victor Chernozhukov and Kengo Kato},
  title = {{Uniform post-selection inference for least absolute deviation regression and other Z-estimation problems}},
  booktitle = {Biometrika},
  year = {2014},
  doi = {10.1093/biomet/asu056},
}
Figure