Abstract
We derive an exact p-value for testing a global null hypothesis in a general adaptive regression problem. The general approach uses the Kac-Rice formula, as described in (Adler & Taylor 2007). The resulting formula is exact in finite samples, requiring only Gaussianity of the errors. We apply the formula to the lasso, group lasso, and principal components and matrix completion problems. In the case of the lasso, the new test relates closely to the recently proposed covariance test of Lockhart et al. (2013).
Introduction
Modern statistical modeling often relies on regularized regression techniques, such as the Lasso, to perform simultaneous variable selection and estimation. However, performing valid inference after model selection—known as post-selection inference—is notoriously difficult. Standard p-values are typically invalid because the model itself is chosen based on the data, a challenge that leads to biased results if not properly addressed.
This paper addresses the global null hypothesis: testing whether the true underlying signal is zero when the first feature (or group) enters the model. While previous breakthroughs like the "covariance test" provided asymptotic approximations for the Lasso, these can be overly conservative or fail in finite samples. We introduce a method to calculate exact, finite-sample p-values for a wide variety of penalties, including the Lasso, Group Lasso, and Nuclear Norm, by leveraging the Kac-Rice formula.
Methodology: Geometry and the Kac-Rice Formula
The core innovation lies in reframing the statistical problem of finding the first "knot" in a regularization path as a geometric problem involving Gaussian processes.
- Gaussian Process Maximization: The entry of the first variable into a model is mathematically equivalent to finding the maximum of a Gaussian process restricted to a specific convex set defined by the penalty's dual norm.
- The Kac-Rice Formula: To determine the distribution of this maximum, we apply the Kac-Rice formula from random field theory. This formula is traditionally used to count the global maximizers of a smooth Gaussian field.
- Exact Pivotal Quantity: By conditioning on the selection event (which variable was chosen) and the local geometry (tangent spaces and Hessians), we derive a conditional survival function. This function acts as an exact "pivot," guaranteed to follow a Uniform(0,1) distribution under the null hypothesis.
- Computation: The authors demonstrate that the required geometric quantities can be reliably computed by solving specific convex optimization problems using standard solvers like ADMM.
Empirical Validation
The proposed test was validated using various simulated designs, including "fat" matrices (more features than samples), highly correlated features, and nested groups. The classic Diabetes dataset was also used as a base design matrix for several simulations to ensure the test holds up under real-world feature correlation structures.
Key Results
The Kac-Rice test demonstrates superior performance compared to existing asymptotic methods across several dimensions:
- Perfect Calibration: Across all tested scenarios—Lasso, Group Lasso, PCA, and Matrix Completion—the test produces perfectly uniform p-values under the null hypothesis, indicating an exact test.
- Comparison with the Covariance Test: Empirical results show that the older covariance test's exponential approximation is often flawed outside simple cases. It tends to be conservative for the Lasso and wildly inaccurate for more complex penalties like the Nuclear Norm. The Kac-Rice test provides a robust fix for these issues.
Limitations and Future Work
While the Kac-Rice framework is powerful, it has several limitations:
- Gaussian Assumption: The mathematical derivation relies heavily on normally distributed errors. While robust for Lasso, the test for Nuclear Norm penalties can break down under non-Gaussian noise.
- Computational Complexity: Calculating geometric limits for complex penalties involves large eigenvalue problems, which may be expensive for very large datasets.
- Scope: Currently, the test focuses on the global null (the first knot). Extending this to subsequent knots in the regularization path (e.g., testing if remaining unselected variables are zero) is a primary goal for future research.
Summary updated