Abstract
We develop a framework for inference after model selection based on the Lasso. At the core of this framework is a result that characterizes the exact (non-asymptotic) distribution of a pivot computed from the Lasso solution. This pivot allows us to (i) devise a test statistic that has an exact (non-asymptotic) $\text{Unif}(0,1)$ distribution under the null hypothesis that all relevant variables have been included in the model, and (ii) construct valid confidence intervals for the selected coefficients that account for the selection procedure.
Introduction
In modern statistical practice, researchers often face high-dimensional datasets where the number of predictors $p$ exceeds the number of observations $n$. A common workflow involves using the Lasso (Least Absolute Shrinkage and Selection Operator) to select a sparse subset of variables, followed by standard ordinary least squares (OLS) regression on those variables to determine significance.
However, this two-stage approach is statistically flawed. Standard inference assumes that the model structure is fixed independently of the data. When the same data is used for both model selection and hypothesis testing, "selection bias" occurs. Variables that appear significant may simply be artifacts of the specific sample, leading to "dishonest" p-values and overly narrow confidence intervals. This paper provides a mathematical framework to perform exact, non-asymptotic inference that explicitly accounts for the Lasso selection process.
Methodology
The Polyhedral Lemma
The authors' primary contribution is the characterization of the Lasso selection event as a geometric constraint. They prove that the event of the Lasso selecting a specific set of variables with a specific sign pattern is equivalent to the response vector $y$ falling into a polyhedron. This polyhedron is defined by a set of linear (affine) constraints:
$$Ay \le b$$
By identifying these constraints, the selection process—often viewed as a "black box"—is transformed into a well-defined conditional probability problem.
Truncated Gaussian Distribution and the Pivot
Assuming the noise in the data follows a Gaussian distribution, conditioning on the selection event is equivalent to restricting the Gaussian distribution to the interior of the defined polyhedron. The authors derive a "pivotal quantity" based on the cumulative distribution function (CDF) of this truncated Gaussian.
This pivot follows a $\text{Unif}(0, 1)$ distribution regardless of the true parameter values, provided the null hypothesis holds. By inverting this test, the framework can produce confidence intervals that maintain their nominal coverage (e.g., 95%) even after the selection has taken place. Unlike previous methods that rely on large-sample asymptotics, this result is exact for any sample size $n$ and any fixed penalty parameter $\lambda$.
Empirical Validation
Synthetic Simulations
To verify the framework, the authors conducted high-dimensional simulations where the ground truth was known. Under the null hypothesis (where no variables are relevant), the adjusted p-values were found to be perfectly distributed between 0 and 1. In contrast, unadjusted p-values were heavily skewed toward zero, which would lead to a high rate of false discoveries in a real-world setting.
The Diabetes Dataset
The method was applied to the classic Diabetes dataset (442 patients, 10 predictors). The analysis highlighted the "price" of selection. For instance, the variable hdl (high-density lipoproteins) yielded an unadjusted p-value of $0.0019$. However, after adjusting for the fact that hdl was selected by the Lasso from a larger pool of candidates, the p-value rose to $0.05$. This shift demonstrates how standard methods can overstate the certainty of a discovery by ignoring the search process that preceded the test.
Discussion and Limitations
While the framework provides a rigorous solution to post-selection inference, it introduces certain trade-offs. Adjusted confidence intervals are generally wider than unadjusted ones, reflecting the inherent uncertainty introduced by using the data twice.
Several limitations remain for future research:
- Known Variance: The current derivation assumes the noise variance $\sigma^2$ is known. While high-dimensional estimators can be used, the inference is only truly "exact" if the variance is specified correctly.
- Boundary Sensitivity: If the data vector $y$ falls near the boundary of the selection polyhedron, the resulting confidence intervals can become extremely wide or even infinite.
- Fixed Penalty: The framework assumes $\lambda$ is fixed, whereas in practice, $\lambda$ is often chosen via cross-validation.
Despite these constraints, the polyhedral approach offers a versatile foundation that can be extended to other selection procedures, such as the Group Lasso, Graphical Lasso, and forward stepwise regression, ensuring that statistical inference remains honest in the age of automated model selection.
Summary updated