Estimation and testing for partially linear single-index models

Hua Liang, Xiang Liu, Runze Li, Chih‐Ling TsaiThe Annals of Statistics 2010 · DOI

TL;DR

This paper introduces a new statistical framework for analyzing data where a response variable depends on a combination of linear and non-linear relationships with predictor variables. They develop efficient methods for estimating model parameters, selecting relevant variables, and testing hypotheses about the model structure, offering a more robust approach than existing single-index models.

What Problem Does This Paper Address?

Many real-world phenomena involve complex relationships between predictors and outcomes. While standard linear regression assumes a simple additive relationship, and single-index models capture non-linearities through a single linear combination of predictors, neither fully captures situations where both linear and non-linear effects are present simultaneously. This paper addresses this gap by proposing the "partially linear single-index model" (PLSIM), which allows for a linear component and a non-linear component, where the non-linear part is a function of a linear combination of some predictors. This model is more flexible and can better represent complex data structures. A key challenge in such models is simultaneously estimating the unknown parameters of both the linear and non-linear parts, selecting which predictors are truly important (variable selection), and testing the validity of the model assumptions.

Methods

The authors propose a two-pronged approach:

  • Profile Least-Squares (PrLS) Estimators: They adapt the profile likelihood approach to estimate the unknown parameters. This involves iteratively estimating the non-linear component and then using these estimates to refine the linear component's estimates. This method aims to achieve "semiparametrically efficient" estimators, meaning they achieve the best possible statistical precision given the model structure. They also show that their PrLS estimators are consistent and asymptotically normal.
  • SCAD (Smoothly Clipped Absolute Deviation) Penalty for Variable Selection: To address the problem of unknown model structure and the need for variable selection, they integrate the SCAD penalty into the PrLS framework. SCAD is a non-convex penalty function known for its ability to perform variable selection and produce "oracle properties" – meaning the selected model performs as well as if the true model were known beforehand. They develop penalized PrLS estimators using SCAD and demonstrate their consistency and oracle properties.
  • BIC (Bayesian Information Criterion) for Tuning Parameter Selection: For the SCAD method, tuning parameters are crucial. They propose a BIC-based selector to consistently identify the true model structure.
  • Hypothesis Testing: The paper also introduces two types of tests:
  • A test for the parametric (linear) coefficients.
  • A goodness-of-fit test for the non-parametric (non-linear) component. Both tests are shown to have desirable asymptotic properties (chi-squared distribution under the null hypothesis).

Datasets

Not discussed in the paper. The paper focuses on theoretical development and simulation studies.

Key Insights and Ideas

  • Bridging Linear and Non-linear Models: The PLSIM offers a powerful way to model data that exhibits both additive linear effects and complex non-linear dependencies, which are common in many scientific fields.
  • Efficient Estimation without Iteration: The proposed profile least-squares approach aims to provide efficient estimators without requiring the computationally intensive iterative procedures often associated with semiparametric models.
  • SCAD for Robust Variable Selection: Integrating SCAD with PrLS provides a method that not only estimates parameters efficiently but also robustly selects relevant predictors, leading to simpler and more interpretable models.
  • Oracle Property: The SCAD estimators achieve the "oracle property," meaning their performance is asymptotically equivalent to knowing the true model structure in advance, which is a strong theoretical guarantee.
  • Consistent Model Selection: The BIC criterion is shown to reliably identify the correct set of predictor variables, which is crucial for building accurate and interpretable models.
  • Comprehensive Testing Framework: The development of both parametric coefficient tests and non-parametric goodness-of-fit tests provides a complete toolkit for validating the proposed PLSIM.

Key Results

  • Efficient Estimators: The profile least-squares estimators are shown to be semiparametrically efficient and possess asymptotic normality.
  • Consistent Variable Selection: The SCAD estimators are demonstrated to be consistent and achieve the oracle property, meaning they correctly identify the important variables with high probability.
  • Model Selection Performance: The BIC selector is shown to consistently identify the true model.
  • Hypothesis Test Performance: Simulation studies indicate that the proposed hypothesis tests control the Type I error rate (false positive rate) well and have good power to detect deviations from the null hypothesis.
  • Simulation Studies: Monte Carlo simulations on generated data demonstrate that the proposed methods perform well in finite samples, with accuracy improving as sample size increases. The PrLS method is shown to be competitive with or superior to existing methods like MAVE.

Limitations and Future Work

  • Computational Cost: While the PrLS method aims to reduce computational burden compared to some iterative methods, the SCAD optimization can still be computationally intensive, especially with high-dimensional data.
  • Tuning Parameter Sensitivity: The performance of SCAD relies on the choice of tuning parameters. While BIC is proposed for selection, its performance can still be sensitive to the data.
  • Future Work: The authors suggest several avenues for future research:
  • Extending the results to partially linear multiple-index models.
  • Developing SCAD estimators for generalized partially linear single-index models.
  • Investigating model selection for partially linear single-index models with error-prone covariates.

Summary updated

Abstract

In partially linear single-index models, we obtain the semiparametrically efficient profile least-squares estimators of regression coefficients. We also employ the smoothly clipped absolute deviation penalty (SCAD) approach to simultaneously select variables and estimate regression coefficients. We show that the resulting SCAD estimators are consistent and possess the oracle property. Subsequently, we demonstrate that a proposed tuning parameter selector, BIC, identifies the true model consistently. Finally, we develop a linear hypothesis test for the parametric coefficients and a goodness-of-fit test for the nonparametric component, respectively. Monte Carlo studies are also presented.
BibTeX
@article{liang2010estimation,
  author = {Hua Liang and Xiang Liu and Runze Li and Chih‐Ling Tsai},
  title = {{Estimation and testing for partially linear single-index models}},
  journal = {The Annals of Statistics},
  year = {2010},
  doi = {10.1214/10-aos835},
}
Figure