Variable selection in semiparametric regression modeling

Runze Li, Hua LiangThe Annals of Statistics 2008 · DOI

TL;DR

This paper introduces novel methods for variable selection in semiparametric regression models, tackling the challenge of simultaneously identifying important variables for both parametric and nonparametric components. The proposed techniques, using nonconcave penalized likelihood and a generalized likelihood ratio test, offer an efficient and theoretically sound approach that performs comparably to an "oracle" (ideal) procedure.

What Problem Does This Paper Address?

In statistical modeling, selecting the right variables is crucial for building accurate and interpretable models. When dealing with semiparametric regression models, which combine the flexibility of nonparametric methods with the interpretability of parametric ones, variable selection becomes significantly more complex. Traditional methods like stepwise regression or best subset selection, when applied to semiparametric models, require separate model selection for the nonparametric parts for each potential submodel. This leads to an enormous computational burden, making these approaches impractical for many real-world scenarios. This paper aims to bridge this gap by developing efficient variable selection procedures for semiparametric regression that can handle both parametric and nonparametric components simultaneously.

Methods

The core of the paper's contribution lies in two main areas:

  • Parametric Component Variable Selection: The authors propose a class of variable selection procedures for the parametric part of the model using nonconcave penalized likelihood. This approach, inspired by Fan and Li's work on parametric models, involves adding a penalty term to the likelihood function that encourages sparsity (i.e., driving some coefficients to exactly zero). They introduce the Smoothly Clipped Absolute Deviation (SCAD) penalty as a key example, which offers desirable properties like oracle properties (meaning the selected model behaves as if the true model were known). The method aims to simultaneously select significant variables and estimate their coefficients, overcoming the computational issues of traditional methods.
  • Nonparametric Component Variable Selection: For the nonparametric component, the paper extends the generalized likelihood ratio test (GLRT) from fully nonparametric models to semiparametric settings. This allows for the selection of significant variables within the nonparametric functions. A key finding is that the proposed GLRT exhibits a "Wilks phenomenon," meaning its limiting null distribution follows a chi-square distribution that is independent of nuisance parameters, simplifying inference.

Datasets

The paper uses simulated data for its extensive Monte Carlo studies. It also applies the proposed methodology to a real-world dataset from the General Hospital Burn Center at the University of Southern California, concerning survival probabilities of burn victims.

Key Insights and Ideas

  • Unified Approach: The paper successfully unifies variable selection for both parametric and nonparametric components within a single framework, a significant advancement for semiparametric modeling.
  • Computational Efficiency: By avoiding the need for separate model selection at each step for the nonparametric parts, the proposed methods drastically reduce computational burden compared to traditional approaches.
  • Oracle Properties: The theoretical results demonstrate that the proposed penalized likelihood estimator achieves "oracle properties," meaning it performs asymptotically as well as if the true model were known. This is a strong theoretical guarantee for the quality of the selected model.
  • Wilks Phenomenon for GLRT: The extension of the GLRT to semiparametric models and the discovery of its chi-square distribution with a nuisance-parameter-free limiting distribution is a significant theoretical contribution, simplifying hypothesis testing.

Key Results

  • Theoretical Guarantees: The paper establishes the rate of convergence and asymptotic normality for the proposed estimators.
  • Simulation Performance: Extensive Monte Carlo simulations show that the SCAD penalty-based procedure performs very well, often close to an "oracle" procedure, and significantly outperforms traditional methods like AIC, BIC, and LASSO in terms of model error and complexity.
  • Computational Speed: The penalized likelihood methods (SCAD and L1) are shown to be substantially faster than best subset selection, especially as the dimensionality of the problem increases.
  • Real Data Application: The methodology is successfully applied to a burn victim survival dataset, identifying significant covariates and demonstrating the practical utility of the approach. For instance, the GLRT identified prior respiratory disease as a significant factor in survival.

Limitations and Future Work

  • Bandwidth Selection: While the paper discusses bandwidth selection for the nonparametric component, it notes that achieving the necessary conditions for oracle properties requires specific bandwidth orders (e.g., $nh^4 \to 0$ and $nh^2/\log(n)^2 \to \infty$). The paper suggests that data-driven methods or asymptotic theory can be used to select such bandwidths.
  • Multivariate U: The paper mentions that the methods can be extended to multivariate U (nonparametric covariates), but this extension might not be very useful in practice due to the "curse of dimensionality."
  • Future Work: The paper's success in developing these procedures suggests potential for further exploration in more complex semiparametric settings and for developing more robust data-driven selection mechanisms for tuning parameters.

Summary updated

Abstract

In this paper, we are concerned with how to select significant variables in semiparametric modeling. Variable selection for semiparametric regression models consists of two components: model selection for nonparametric components and select significant variables for parametric portion. Thus, it is much more challenging than that for parametric models such as linear models and generalized linear models because traditional variable selection procedures including stepwise regression and the best subset selection require model selection to nonparametric components for each submodel. This leads to very heavy computational burden. In this paper, we propose a class of variable selection procedures for semiparametric regression models using nonconcave penalized likelihood. The newly proposed procedures are distinguished from the traditional ones in that they delete insignificant variables and estimate the coefficients of significant variables simultaneously. This allows us to establish the sampling properties of the resulting estimate. We first establish the rate of convergence of the resulting estimate. With proper choices of penalty functions and regularization parameters, we then establish the asymptotic normality of the resulting estimate, and further demonstrate that the proposed procedures perform as well as an oracle procedure. Semiparametric generalized likelihood ratio test is proposed to select significant variables in the nonparametric component. We investigate the asymptotic behavior of the proposed test and demonstrate its limiting null distribution follows a chi-squared distribution, which is independent of the nuisance parameters. Extensive Monte Carlo simulation studies are conducted to examine the finite sample performance of the proposed variable selection procedures.
BibTeX
@article{li2008variable,
  author = {Runze Li and Hua Liang},
  title = {{Variable selection in semiparametric regression modeling}},
  journal = {The Annals of Statistics},
  year = {2008},
  doi = {10.1214/009053607000000604},
}
Figure