Uniform Post Selection Inference for LAD Regression Models

Alexandre Belloni, Victor Chernozhukov, Kengo Kato

Introduction

In the era of high-dimensional data, researchers frequently encounter the "large $p$, small $n$" problem, where the number of potential predictors ($p$) exceeds the number of observations ($n$). The standard statistical workflow involves a model selection step—often using a tool like Lasso—followed by inference on the selected variables. However, this "naive" post-selection inference is frequently invalid. If a variable with a small but non-zero effect is omitted during selection, it biases the coefficients of the remaining variables, leading to unreliable p-values and confidence intervals.

This paper, published in Biometrika (2014), addresses this challenge by developing a framework for uniformly valid confidence regions in high-dimensional sparse Least Absolute Deviation (LAD) regression models. LAD, or median regression, is highly valued for its robustness to outliers and non-normal noise, but it presents significant analytical challenges due to its non-smooth objective function.

The Problem: The Failure of Naive Selection

The fundamental issue with standard post-selection methods is the lack of "uniformity." Most existing methods rely on the "oracle property," assuming that the model selection step is perfect. In reality, selection mistakes are common, particularly when coefficients are "local to zero"—too small to be reliably detected by Lasso but large enough to cause omitted-variable bias.

In these scenarios, the coverage of confidence intervals can drop significantly below the nominal level (e.g., a 95% interval might only contain the true value 50% of the time). This paper seeks to provide an estimator that remains accurate even when the initial model selection step makes moderate mistakes.

Methodology: Neyman’s Orthogonalization

The authors introduce a three-step "instrumental LAD" estimation procedure designed to be "immunized" against selection errors. The core of this approach is Neyman’s Orthogonalization, which ensures that the estimating equation is insensitive to small errors in the estimation of nuisance parameters (the effects of control variables not currently under test).

The Three-Step Procedure:

  1. Nuisance Estimation: The "nuisance" part of the regression—the effects of all control variables—is estimated using $\ell_1$-penalized LAD or a post-selection variant.
  2. Instrument Construction: An "instrumental variable" is constructed for the regressor of interest. This involves running a Lasso regression of the target variable on all other controls to isolate the portion of the target variable that is orthogonal to the others.
  3. Orthogonalized Estimation: The target coefficient is found by solving a moment equation. Because of the orthogonalization, the derivative of this equation with respect to the nuisance parameters is zero, preventing selection errors from contaminating the final estimate.

Theoretical and Empirical Results

The paper establishes that in homoscedastic regression models, the instrumental LAD estimator is asymptotically root-$n$ normal, and this normality holds uniformly across the underlying sparse model. This means the resulting confidence regions are valid regardless of whether the underlying coefficients are large, small, or zero.

Monte Carlo Evidence

The authors evaluated the method using extensive simulations with $n=250$ and $p=300$. Across 100 different experimental designs varying in correlation and signal strength, the results were conclusive:

  • Uniform Validity: The proposed method consistently maintained a 95% coverage rate.
  • Naive Breakdown: Standard post-selection methods performed poorly, with rejection probabilities deviating sharply from nominal levels. In highly correlated designs, the bias in naive methods rendered their confidence intervals essentially useless.
  • Efficiency: The estimator was shown to be semi-parametrically efficient, utilizing available information as effectively as any regular estimator.

Conclusion and Limitations

This research provides a robust solution for inference in high-dimensional median regression. By moving beyond the "oracle" assumption and focusing on uniform validity, the authors offer a framework that is more reliable for real-world data analysis where model selection is rarely perfect.

However, some limitations remain. The current proofs primarily assume homoscedasticity (constant noise levels), and while the authors suggest paths for heteroscedastic cases, estimating the necessary weights remains a delicate task. Furthermore, the method still relies on the "sparsity" assumption—that only a small subset of variables ($s \ll n$) truly drive the regression function. Despite these constraints, the instrumental LAD approach represents a significant step forward in making high-dimensional statistics more rigorous and dependable.

Summary updated

Abstract

We develop uniformly valid confidence regions for a regression coefficient in a high-dimensional sparse LAD (least absolute deviation or median) regression model. The setting is one where the number of regressors p could be large in comparison to the sample size n, but only s « n of them are needed to accurately describe the regression function. Our new methods are based on the instrumental LAD regression estimator that assembles the optimal estimating equation from either post ℓ- penalised LAD regression or ℓ1- penalised LAD regression. The estimating equation is immunised against non-regular estimation of nuisance part of the regression function, in the sense of Neyman. We establish that in a homoscedastic regression model, under certain conditions, the instrumental LAD regression estimator of the regression coefficient is asymptotically root-n normal uniformly with respect to the underlying sparse model. The resulting confidence regions are valid uniformly with respect to the underlying model. The new inference methods outperform the naive, 'oracle based' inference methods, which are known to be not uniformly valid- with coverage property failing to hold uniformly with respect the underlying model- even in the setting with p = 2. We also provide Monte-Carlo experiments which demonstrate that standard post-selection inference breaks down over large parts of the parameter space, and the proposed method does not.
BibTeX
@inproceedings{belloni2014uniform,
  author = {Alexandre Belloni and Victor Chernozhukov and Kengo Kato},
  title = {{Uniform Post Selection Inference for LAD Regression Models}},
  booktitle = {Biometrika},
  year = {2014},
  doi = {10.1093/biomet/asu056},
}
Figure