Valid Post-Selection Inference in High-Dimensional Approximately Sparse Quantile Regression Models

Alexandre Belloni, Victor Chernozhukov, Kengo Kato

Introduction

In the modern landscape of statistical analysis, researchers frequently encounter high-dimensional datasets where the number of potential predictors ($p$) exceeds the sample size ($n$). To handle this complexity, model selection techniques like Lasso are often employed to identify a sparse subset of relevant variables. However, a significant challenge arises: "naive" inference conducted after model selection typically fails to account for the selection process itself. This leads to biased estimates and invalid confidence intervals—a phenomenon known as the post-selection inference problem.

This paper introduces a robust framework for performing valid inference in high-dimensional quantile regression models. By utilizing "orthogonal score functions," the proposed methods provide a mathematical safety net that remains valid even when the initial model selection step makes moderate mistakes, such as overlooking variables with small but non-zero effects.

The Problem: Post-Selection Bias in Quantile Regression

Standard statistical tests assume that the model structure is fixed before the data is observed. When the same data is used both to select variables and to test their significance, the resulting p-values are often misleadingly small. This issue is particularly acute in quantile regression, which models specific points in the conditional distribution (e.g., the 10th or 90th percentile) rather than just the mean.

In quantile regression, omitting a relevant variable—even one with a small coefficient—can lead to severe "omitted variable bias." Traditional high-dimensional theory often relies on a "beta-min" condition, which assumes all non-zero effects are large enough to be easily detected. If this assumption is violated, standard selection tools drop these "small" variables, ruining the final estimate's reliability.

Proposed Methodology

The authors propose two primary algorithmic approaches to ensure valid inference, both centered on the concept of Neyman-orthogonal score functions.

1. Orthogonal Score Function (Algorithm 2.1)

This method constructs a score function—the mathematical requirement used to identify the estimate—that is intentionally designed to be "immune" to first-order errors in estimating nuisance parameters (the other variables in the model). By making the score function locally insensitive to these estimation errors, the resulting estimator for the target coefficient remains robust.

2. Weighted Double Selection (Algorithm 2.2)

This approach provides a more intuitive "belt and suspenders" strategy for variable selection:

  • Step 1: Select variables that are predictive of the outcome ($y$).
  • Step 2: Select variables that are predictive of the treatment or policy variable of interest ($d$).
  • Step 3: Perform the final quantile regression using the union of variables from both steps.

By selecting variables that correlate with both the treatment and the outcome, the researcher ensures that potential confounders are retained in the model, even if their individual effect on the outcome was too small to be caught in a single-step selection.

Theoretical Contributions and Uniform Validity

A key technical achievement of this work is establishing the uniform validity of the proposed confidence regions. Unlike previous methods that only work under specific, narrow conditions, these methods are "honest" in the sense that they provide reliable coverage across a wide range of parameter values, including cases where the model is only "approximately sparse" (i.e., many variables have very small, but non-zero, effects).

Furthermore, the authors developed new techniques for estimating the conditional density function in high dimensions. This is a notoriously difficult task required for quantile regression but essential for calculating the standard errors of the estimates when $p > n$.

Empirical Evidence

Monte Carlo Simulations

The authors conducted extensive simulations across 100 different experimental designs, varying correlation strengths and noise types. The results were stark: while naive selection methods often reported rejection rates as high as 50% for a 5% significance test (indicating a massive overstatement of confidence), the proposed Double Selection and Orthogonal Score methods maintained the correct 5% error rate.

Case Study: Childhood Malnutrition in India

The framework was applied to the Demographic and Health Surveys (DHS) data, analyzing a sample of 37,649 children from India. The goal was to identify risk factors for stunted growth by modeling the 10th percentile of child height against 30 socio-economic and biological factors.

A naive Lasso approach dropped 16 of the 30 variables, including critical factors like mother's BMI and age. In contrast, the authors' robust method retained these variables, demonstrating that they were indeed statistically significant at the 10th percentile. This highlights the method's ability to provide a more nuanced understanding of how policies affect the "tails" of a distribution—such as the most vulnerable, underweight children.

Conclusion

This research provides a rigorous foundation for inference in high-dimensional quantile regression. By moving beyond the restrictive "beta-min" assumptions and providing a framework robust to moderate model selection mistakes, the authors offer a practical tool for researchers working with complex, large-scale datasets. While the method assumes "approximate sparsity" and carries a higher computational cost due to the need for repeated selection steps, it ensures that the resulting scientific conclusions are statistically sound and reproducible.

Summary updated

Abstract

This work proposes new inference methods for a regression coefficient of interest in a (heterogeneous) quantile regression model. We consider a high-dimensional model where the number of regressors potentially exceeds the sample size but a subset of them suffice to construct a reasonable approximation to the conditional quantile function. The proposed methods are (explicitly or implicitly) based on orthogonal score functions that protect against moderate model selection mistakes, which are often inevitable in the approximately sparse model considered in the present paper. We establish the uniform validity of the proposed confidence regions for the quantile regression coefficient. Importantly, these methods directly apply to more than one variable and a continuum of quantile indices. In addition, the performance of the proposed methods is illustrated through Monte-Carlo experiments and an empirical example, dealing with risk factors in childhood malnutrition.
BibTeX
@article{belloni2018valid,
  author = {Alexandre Belloni and Victor Chernozhukov and Kengo Kato},
  title = {{Valid Post-Selection Inference in High-Dimensional Approximately Sparse Quantile Regression Models}},
  journal = {Journal of the American Statistical Association},
  year = {2018},
  doi = {10.1080/01621459.2018.1442339},
}
Figure