Partial Consistency with Sparse Incidental Parameters

Jianqing Fan, Runlong Tang, Xiaofeng Shi

TL;DR

This paper introduces a penalized estimation method for linear regression models with both structural and sparse incidental parameters. It demonstrates a novel "partial consistency" phenomenon where structural parameters are consistently estimated, but incidental ones are not, offering a robust approach for complex high-dimensional data.

What Problem Does This Paper Address?

In high-dimensional statistical modeling, we often encounter situations where some parameters are of primary interest (structural parameters) and others are specific to individual data points or groups and are not expected to generalize (incidental parameters). Traditional methods struggle when these incidental parameters are numerous and high-dimensional, especially when they are also sparse (meaning most of them are zero). This paper tackles the challenge of estimating structural parameters effectively in such settings, where standard penalized methods, successful for structural-only models, fall short due to the presence of these numerous incidental parameters. The core problem is how to disentangle the effects of structural parameters from the noise introduced by a large number of sparse incidental parameters.

Methods

The authors propose a penalized estimation approach that specifically targets the sparse incidental parameters. The key idea is to apply a penalty function (like LASSO) to the incidental parameters, encouraging sparsity and thus identifying which ones are truly non-zero. They derive theoretical guarantees for the consistency and asymptotic normality of the structural parameter estimators, showing they possess an "oracle property" (meaning they perform as well as if the true sparsity pattern were known).

A novel finding is the "partial consistency" phenomenon: while the structural parameters are consistently estimated, the incidental parameters are not, exhibiting only partial selection consistency. To further improve statistical inference, they introduce a two-step penalized estimator. This method first identifies and effectively "removes" the influence of large incidental parameters before estimating the structural parameters, leading to fewer asymptotic distributions and better suitability for constructing confidence intervals. The paper also extends these methods to scenarios where the number of structural parameters grows with the sample size and provides a data-driven approach for selecting the regularization parameter.

Datasets

The paper evaluates its methods using simulated data generated under various scenarios to mimic different types and magnitudes of incidental parameters. They also analyze a real dataset related to genome-wide association studies (GWAS) investigating the association between gene expression levels and SNPs.

Key Insights and Ideas

  • Partial Consistency: The most striking insight is the discovery of "partial consistency." It's counterintuitive that structural parameters can be reliably estimated while incidental ones cannot, yet this is precisely what the proposed method achieves. This highlights the method's ability to isolate and estimate the generalizable effects from data-specific noise.
  • Penalization for Incidental Parameters: Applying penalization directly to incidental parameters, rather than just structural ones, is a novel strategy that effectively handles their sparsity and high dimensionality.
  • Two-Step Estimation for Robustness: The two-step estimator offers a more stable and efficient approach for statistical inference, particularly when incidental parameters have a substantial impact. It addresses the limitations of the one-step method by mitigating the influence of large incidental parameters.
  • Data-Driven Regularization: The proposed data-driven method for selecting the regularization parameter is crucial for practical application, as optimal parameter choices are often unknown in real-world scenarios.

Key Results

  • The penalized estimator for structural parameters is shown to be consistent and asymptotically Gaussian, exhibiting an oracle property.
  • Penalized estimators for incidental parameters achieve partial selection consistency but not full consistency.
  • The two-step penalized estimator provides improved statistical inference properties compared to the one-step estimator, especially in challenging cases with large incidental parameters.
  • Simulations demonstrate that the penalized methods significantly outperform ordinary least squares (OLS) in terms of mean squared error, especially when incidental parameters are large.
  • The data-driven regularization parameter selection method performs well, yielding results comparable to using optimal parameters.
  • The real data analysis shows the method's applicability in a complex biological context, estimating associations between gene expression and SNPs.

Limitations and Future Work

The authors acknowledge that the theoretical analysis for the penalized estimators of incidental parameters shows only partial selection consistency, not full consistency. They also note that the convergence rate of the one-step estimator can be suboptimal in cases with large incidental parameters. Future work could explore more sophisticated data-driven regularization techniques and investigate the performance of these methods in more complex model settings beyond linear regression. The paper also suggests extending the methodology to situations where the structural parameter dimension diverges faster than the sample size.

Summary updated

Abstract

Penalized estimation principle is fundamental to high-dimensional problems. In the literature, it has been extensively and successfully applied to various models with only structural parameters. As a contrast, in this paper, we first apply this penalization principle to a linear regression model with a finite-dimensional vector of structural parameters and a high-dimensional vector of sparse incidental parameters. For the estimators of the structural parameters, we derive their consistency and asymptotic normality, which reveals an oracle property. However, the penalized estimators for the incidental parameters possess only partial selection consistency but not consistency. This is an interesting partial consistency phenomenon: the structural parameters are consistently estimated while the incidental ones cannot. For the structural parameters, also considered is an alternative two-step penalized estimator, which has fewer possible asymptotic distributions and thus is more suitable for statistical inferences. We further extend the methods and results to the case where the dimension of the structural parameter vector diverges with but slower than the sample size. A data-driven approach for selecting a penalty regularization parameter is provided. The finite-sample performance of the penalized estimators for the structural parameters is evaluated by simulations and a real data set is analyzed.
BibTeX
@inproceedings{fan2017partial,
  author = {Jianqing Fan and Runlong Tang and Xiaofeng Shi},
  title = {{Partial Consistency with Sparse Incidental Parameters}},
  booktitle = {Statistica},
  year = {2017},
  doi = {10.5705/ss.202017.0027},
}
Figure