Inference Without Compatibility

Michael Law, Ya’acov RitovarXiv 2019 · DOI · arXiv

Abstract

In high-dimensional linear models, statistical inference typically relies on the "compatibility condition"—a restrictive technical assumption on the design matrix—to ensure the reliability of estimators like the Lasso. This paper demonstrates that this condition is not a fundamental requirement for valid inference. By employing an exponential weighting approach to aggregate models, the authors develop the first $\sqrt{n}$-consistent estimators for low-dimensional coefficients, signal strength, and noise levels without requiring the design matrix to be Lasso-friendly. Theoretical results are supported by numerical simulations demonstrating superior coverage and efficiency compared to traditional de-biased Lasso methods, particularly in settings with high predictor correlation.

Introduction: The High-Dimensional Challenge

In modern data science, we frequently encounter high-dimensional linear models where the number of predictors ($p$) far exceeds the number of observations ($n$). To extract meaningful insights from such data, researchers generally rely on the assumption of sparsity—the idea that only a small subset of predictors significantly influences the response variable.

The most prominent tool for navigating this landscape is the Lasso (Least Absolute Shrinkage and Selection Operator). However, while the Lasso is excellent for prediction and variable selection, its use in rigorous statistical inference (generating p-values and confidence intervals) is often tethered to a "compatibility condition." This condition essentially mandates that the predictors are not too highly correlated, preventing the model from becoming "confused" by collinearity. As this paper argues, this condition is frequently an artifact of the mathematical proofs used for the Lasso rather than a fundamental limit of the statistical problem itself.

Methodology: Aggregation over Selection

To bypass the limitations of the Lasso, the authors shift from a model selection paradigm to a model aggregation paradigm using Exponential Weighting (EW).

The Three-Stage Procedure

For estimating a specific coefficient of interest ($\beta$), the researchers propose a robust three-stage approach:

  1. Response Prediction: Predict the response $Y$ using the high-dimensional covariates $Z$.
  2. Covariate Prediction: Predict the specific covariate of interest $X$ using the remaining covariates $Z$.
  3. Residual Regression: Perform a simple least-squares regression on the residuals derived from the first two steps.

Exponential Weighting and MCMC

Instead of selecting a single "best" model, the EW approach aggregates over all possible models of a specific size. Each model is assigned a weight proportional to $\exp(-\text{Residual Sum of Squares} / \alpha)$. This method, inspired by the work of Leung & Barron (2006), is known to maintain prediction consistency even under highly unfavorable design matrices.

Because summing over all possible model combinations is computationally NP-hard, the authors implement a Metropolis-Hastings algorithm. By utilizing a random walk over a Johnson graph, they can practically approximate these exponential weights, making the theoretical estimator computationally tractable.

Key Insights and Theoretical Results

Breaking the Compatibility Barrier

The primary contribution of this work is the proof that the compatibility condition is optional. The authors demonstrate that their estimators for the coefficient ($\beta$), signal strength ($\sigma_\mu^2$), and noise variance ($\sigma_\epsilon^2$) are all $\sqrt{n}$-consistent and asymptotically normal. This represents the "gold standard" rate of convergence in statistics, achieved here without the standard restrictive assumptions on the design matrix.

The Sparsity Trade-off

The paper notes a critical theoretical trade-off: while the compatibility condition can be dropped, it necessitates a slightly stronger version of sparsity. Theorem 6 in the paper illustrates that one cannot simultaneously have "bad" designs (high correlation) and "very weak" sparsity while maintaining $\sqrt{n}$-consistency. This clarifies the fundamental limits of high-dimensional inference.

Efficiency

The proposed EW estimators are shown to attain the semi-parametric efficiency bound. This means that, within this class of high-dimensional problems, these estimators are as precise as theoretically possible.

Empirical Performance

The authors validated their approach through extensive numerical simulations, specifically targeting scenarios designed to break traditional methods. Using a setup of $n=100$ and $p=500$ with an "equi-correlation" structure ($\rho = 0.8$), they compared the EW approach against the standard De-biased Lasso (DLA).

  • Coverage: Under high correlation, the DLA’s coverage for a nominal 95% confidence interval plummeted below 25%. In contrast, the EW estimators maintained near 95% coverage.
  • Robustness: The method proved robust across various noise distributions, including Gaussian, double exponential, and heavy-tailed $t(3)$ distributions.

Limitations and Future Directions

While the EW approach offers a significant theoretical and practical leap, it is not without challenges. The computational burden of the Metropolis-Hastings algorithm, while tractable, is higher than the simple path algorithms used for the Lasso. Furthermore, while the MCMC approach performs well in practice, there is currently no worst-case polynomial-time guarantee for convergence without the restricted eigenvalue condition.

Future research may focus on automating the selection of tuning parameters, such as the model size ($u$) and the temperature ($\alpha$), or developing even more efficient sampling algorithms to handle ultra-high-dimensional datasets.

Summary updated

Abstract

We consider hypotheses testing problems for three parameters in high-dimensional linear models with minimal sparsity assumptions of their type but without any compatibility conditions. Under this framework, we construct the first $\sqrt{n}$-consistent estimators for low-dimensional coefficients, the signal strength, and the noise level. We support our results using numerical simulations and provide comparisons with other estimators.
BibTeX
@misc{law2019inference,
  author = {Michael Law and Ya’acov Ritov},
  title = {{Inference Without Compatibility}},
  year = {2019},
  eprint = {1903.06295},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1903.06295},
}
Figure