Variable selection in nonparametric additive models

Jian Huang, Joël L. Horowitz, Fengrong WeiThe Annals of Statistics 2010 · DOI

TL;DR

This paper introduces the "adaptive group Lasso" for variable selection in nonparametric additive models, particularly useful when dealing with high-dimensional data where many potential predictors might have no effect. The method effectively identifies which predictor variables truly influence the outcome, even when the number of predictors far exceeds the number of observations, and achieves optimal estimation rates.

What Problem Does This Paper Address?

In many real-world scenarios, the relationship between predictors and an outcome isn't linear. Nonparametric additive models offer a flexible way to capture these complex, nonlinear relationships by assuming the outcome is a sum of functions of individual predictors. However, a major challenge arises when we have a vast number of potential predictors (high-dimensional data) and only a limited number of observations. It's crucial to identify which of these predictors actually matter (variable selection) to build parsimonious and interpretable models, and to avoid overfitting. Existing methods often struggle with this, especially when the true number of influential predictors is small but the total number of predictors is very large. This paper tackles the problem of selecting the nonzero additive components in such high-dimensional nonparametric additive models.

Methods

The core of the proposed method is an extension of the Lasso technique, specifically the "adaptive group Lasso." Here's a breakdown:

  • Nonparametric Additive Model: The model assumes the outcome $Y$ can be expressed as $Y = \mu + \sum_{j=1}^p f_j(X_j) + \epsilon$, where $f_j$ are unknown smooth functions of individual predictors $X_j$, and $\epsilon$ is noise.
  • Spline Approximation: To handle the unknown functions $f_j$, they are approximated using B-spline bases. This transforms the problem of selecting functions into selecting groups of coefficients corresponding to these spline bases.
  • Group Lasso: This is a penalized regression technique that encourages sparsity at the group level. In this context, each predictor $X_j$ (and its corresponding spline coefficients) forms a group. The group Lasso aims to shrink the coefficients of entire groups to zero, effectively selecting or discarding entire predictor variables.
  • Adaptive Group Lasso: This is a refinement of the group Lasso. It uses an initial estimate (obtained from the standard group Lasso) to assign different weights to the penalty terms for each group. This adaptive weighting allows for more precise selection, particularly in distinguishing between truly zero and very small but nonzero effects. The authors propose a two-step procedure: first, use group Lasso for initial dimension reduction, and then use adaptive group Lasso for final selection.

Datasets

The paper uses simulated data for its simulation studies. For the data example, they utilize a dataset from Scheetz et al. (2006) concerning gene expression in rat eyes, specifically looking for genes related to TRIM32. This dataset involves 120 observations and 500 potential predictor variables (probe sets).

Key Insights and Ideas

  • Bridging the Gap: The adaptive group Lasso effectively bridges the gap between linear models and complex nonparametric models in high-dimensional settings, offering a principled way to perform variable selection.
  • Group Structure is Key: Recognizing that the spline basis coefficients for each predictor form a natural "group" is crucial. The group Lasso leverages this structure, which is more appropriate than applying a standard Lasso that treats all coefficients individually.
  • Two-Step Power: The two-step approach (initial group Lasso, then adaptive group Lasso) is a powerful strategy. The first step reduces the complexity, and the second step refines the selection with improved theoretical guarantees.
  • Theoretical Guarantees: The paper provides strong theoretical guarantees for the adaptive group Lasso, showing it can consistently select the correct set of nonzero components with high probability and achieve optimal estimation rates, even when the number of predictors is much larger than the sample size.
  • Nonlinearity Matters: The simulation results and data example highlight that ignoring nonlinearity (e.g., by using a linear model) can lead to poor selection results and misleading inferences, underscoring the need for nonparametric approaches.

Key Results

  • Selection Consistency: The adaptive group Lasso is shown to select the correct set of nonzero additive components with probability approaching one as the sample size increases.
  • Optimal Convergence Rates: The method achieves the optimal rate of convergence for nonparametric estimation of additive models.
  • Simulation Performance: Monte Carlo experiments demonstrate that the adaptive group Lasso performs well, achieving high percentages of correct model selection (CS) and inclusion of correct components (IN), outperforming ordinary Lasso and linear models, especially when the underlying relationships are nonlinear. For instance, in one simulation, the adaptive group Lasso achieved CS rates of 65-81%, significantly higher than ordinary Lasso (12-15.75%).
  • Data Example: In the gene expression data example, the adaptive group Lasso provided a more targeted list of relevant genes compared to the ordinary Lasso and linear models, with better performance in terms of residual sum of squares (RSS).

Limitations and Future Work

  • Fixed Number of Nonzero Components: The theoretical results primarily assume that the number of nonzero additive components ($q$) is fixed and does not grow with the sample size. Relaxing this condition would be an important extension.
  • Penalty Parameter Selection: The theoretical results are derived for a fixed sequence of penalty parameters. The paper acknowledges that selecting penalty parameters using data-driven procedures like BIC or EBIC is an important and challenging problem that deserves further investigation.
  • Linear Nonparametric Models: The current work focuses on linear nonparametric additive models. Extending the adaptive group Lasso to more complex generalized nonparametric additive models or other nonparametric models with high-dimensional data is suggested as future work.

Summary updated

Abstract

We consider a nonparametric additive model of a conditional mean function in which the number of variables and additive components may be larger than the sample size but the number of nonzero additive components is "small" relative to the sample size. The statistical problem is to determine which additive components are nonzero. The additive components are approximated by truncated series expansions with B-spline bases. With this approximation, the problem of component selection becomes that of selecting the groups of coefficients in the expansion. We apply the adaptive group Lasso to select nonzero components, using the group Lasso to obtain an initial estimator and reduce the dimension of the problem. We give conditions under which the group Lasso selects a model whose number of components is comparable with the underlying model, and the adaptive group Lasso selects the nonzero components correctly with probability approaching one as the sample size increases and achieves the optimal rate of convergence. The results of Monte Carlo experiments show that the adaptive group Lasso procedure works well with samples of moderate size. A data example is used to illustrate the application of the proposed method.
BibTeX
@article{huang2010variable,
  author = {Jian Huang and Joël L. Horowitz and Fengrong Wei},
  title = {{Variable selection in nonparametric additive models}},
  journal = {The Annals of Statistics},
  year = {2010},
  doi = {10.1214/09-aos781},
}
Figure