Finite- and Large- Sample Inference for Model and Coefficients in High-dimensional Linear Regression with Repro Samples

Peng Wang, Min-Ge Xie, Linjun ZhangarXiv 2022 · DOI · arXiv

TL;DR

This paper introduces a "repro samples" framework to perform statistical inference in high-dimensional linear regression. Unlike traditional methods that rely on large-sample asymptotics, this approach provides valid confidence sets for both the underlying model and its coefficients with provable finite-sample performance guarantees.

What Problem Does This Paper Address?

In modern data science, we often deal with "high-dimensional" problems where the number of variables ($p$) far exceeds the number of observations ($n$). While tools like the Lasso are great at estimating which variables matter, performing inference—quantifying exactly how certain we are about those variables—is notoriously difficult.

Existing methods, such as the "debiased Lasso," face two major hurdles:

  1. Asymptotic Reliance: They usually only work when the sample size is very large ($n \to \infty$). In many real-world fields like genomics, $n$ is small, and these methods often fail to provide the promised coverage.
  2. Model Uncertainty: Most methods provide confidence intervals for coefficients assuming a specific model was selected. They struggle to account for the uncertainty inherent in the model selection process itself. Because the space of possible models is discrete, standard tools like the Central Limit Theorem don't easily apply.

Methods

The authors propose a framework based on repro samples. The core idea is to conduct inference by creating and studying the behavior of artificial samples that mimic the data's original sampling mechanism.

Key components include:

  • Fisher Inversion: To handle the discrete nature of model selection, the authors use an optimization-based "Fisher inversion" technique. By simulating many copies of artificial error terms ("repro copies"), they construct a model candidate set that is guaranteed to contain the true sparse model with high probability.
  • Fisher-Dempster Inversion: This technique is used to construct level-$\alpha$ confidence sets for the model without needing to estimate the regression coefficients or the noise level first. It essentially "inverts" a Borel set of possible error realizations.
  • The Union Approach: To perform inference on regression coefficients while accounting for model uncertainty, the authors construct confidence sets for each model in the candidate set and take their union. This ensures that even if the "best" model is slightly off, the true coefficients remain covered.

Datasets

The authors evaluated their approach using both synthetic and real-world data:

  • Synthetic Models (M1-M5): These simulations covered various challenging scenarios, including extremely high dimensions ($p=1000, n=50$), decaying signal strengths, and cases of "perfect collinearity" where standard eigenvalue conditions required by other methods do not hold.
  • Real-World Data: Glioblastoma gene expression data from The Cancer Genome Atlas (TCGA). The dataset included 428 samples, which the authors screened down to the 1,000 genes most correlated with survival time to test the method in a high-dimensional, real-world setting.

Key Insights and Ideas

  • Finite-Sample Guarantees: The most significant "aha" moment is that valid inference doesn't require $n$ to go to infinity. By using the repro samples framework, the authors prove that coverage holds for any fixed $n$ and $p$, provided enough computational power (repro copies) is used.
  • Bypassing Covariance Estimation: Unlike debiased Lasso, this method does not rely on estimating a high-dimensional covariance matrix (or its inverse), which is a frequent source of error in other pipelines.
  • Complementary Forces: The paper reveals a trade-off: coverage validity can be driven by either a large sample size ($n$) or increased computation ($d$, the number of repro samples). If your dataset is small, you can compensate by running more simulations.
  • Robustness to Non-Gaussianity: The framework naturally extends to non-Gaussian errors (like Cauchy or t-distributions) because it relies on mimicking the error distribution rather than assuming normality for a test statistic.

Key Results

  • Superior Coverage: In simulations where $n$ was small, the repro samples method maintained the target 95% coverage, while the debiased Lasso often dropped to 30-40% coverage for non-zero coefficients.
  • Efficiency: The confidence sets produced were not only more accurate but also 40% shorter on average than those from state-of-the-art debiasing or bootstrap approaches.
  • Biological Relevance: On the TCGA glioblastoma data, the method identified two key genes (TOP1 and NETO2) with high certainty. These genes have established links to glioblastoma in medical literature, whereas the bootstrap-based methods identified several genes with no known association, suggesting they were picking up spurious signals.

Limitations and Future Work

  • Computational Cost for Weak Signals: While the method is generally efficient ($O(dp^3)$), recovering very weak signals in extremely high dimensions may require a very large number of repro samples ($d$), increasing the computational burden.
  • Identifiability: The authors assume a unique "smallest" true model. If multiple models of the same size are perfectly collinear, the method will cover one of them, but providing a joint confidence set for all equivalent "true" models remains an open challenge.
  • Future Directions: The authors suggest extending this framework to Generalized Linear Models (GLMs) (like logistic regression) and robust quantile regression, where the relationship between the error term and the response is more complex.

Summary updated

Abstract

In this paper, we present a novel and effective inference approach to conduct both finite- and large-sample inference for high-dimensional linear regression models. This approach is developed under the so-called repro samples framework, in which we conduct statistical inference by creating and studying the behavior of artificial samples that are obtained by mimicking the sampling mechanism of the data. We construct confidence sets for (a) the true model corresponding to the nonzero coefficients, (b) a single or any collection of regression coefficients, and (c) both the model and regression coefficients jointly. To facilitate the constructions of these confidence sets and overcome computational difficulties of searching all possible models, we use an innovative Fisher inversion technique to construct a model candidate set that includes the true sparse model with the probability close to 1 for models with both Gaussian and non-Gaussian errors. The proposed approach fills in two major gaps in the high-dimensional regression literature: (1) lack of effective approaches to addressing model selection uncertainty and providing valid inference for the underlying true model; (2) lack of effective inference approaches to guaranteeing finite-sample performance. We provide both finite-sample and asymptotic results to theoretically guarantee the performance of the proposed methods. In addition, our numerical results demonstrate that the proposed methods are valid and achieve better coverage with smaller confidence sets than the current state-of-the-art approaches, such as debiasing and bootstrap approaches.
BibTeX
@misc{wang2022finite,
  author = {Peng Wang and Min-Ge Xie and Linjun Zhang},
  title = {{Finite- and Large- Sample Inference for Model and Coefficients in High-dimensional Linear Regression with Repro Samples}},
  year = {2022},
  eprint = {2209.09299},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2209.09299},
}
Figure