Kernel Meets Sieve: Post-Regularization Confidence Bands for Sparse Additive Model

Junwei Lu, Mladen Kolar, Han LiuJournal of the American Statistical Association 2019 · DOI

TL;DR

This paper introduces a novel "kernel-sieve hybrid" method for constructing confidence bands in high-dimensional sparse additive models. This approach effectively bridges the gap between kernel and spline-based estimation techniques, offering a more robust way to quantify uncertainty in complex statistical models.

What Problem Does This Paper Address?

In statistics, we often want to understand the relationship between a target variable (Y) and multiple input variables (X). When this relationship is complex and not easily described by a simple linear equation, we turn to nonparametric methods. Sparse additive models (SpAMs) are a popular choice, assuming the relationship can be broken down into the sum of simpler, individual functions of specific input variables, with the added assumption that only a few of these functions are actually non-zero (sparse).

While estimating these functions has seen significant progress, a crucial aspect that has lagged behind is statistical inference, specifically providing confidence bands. Confidence bands are essential for understanding the uncertainty around our estimates – they give us a range where the true underlying function is likely to lie. Existing methods for confidence bands in nonparametric settings often rely on kernel or local polynomial estimators, while SpAM estimation typically uses sieve estimators. This paper addresses the challenge of constructing reliable confidence bands for SpAMs, especially in high-dimensional settings where the number of input variables is large, by developing a method that leverages the strengths of both kernel and sieve approaches.

Methods

The core contribution is a kernel-sieve hybrid estimator. This method combines:

  • Kernel Regression: Used locally for the component of interest (the function we want to estimate and build a confidence band for). This allows for flexible estimation at specific points.
  • Spline-Based Sieve Estimation: Used for the other components of the additive model. Splines are piecewise polynomial functions that can efficiently approximate complex shapes.
  • Group Lasso Penalty: Applied to shrink coefficients of the spline basis functions, aiding in variable selection and identifying which input variables are truly important.

The authors also introduce a de-biasing method to correct for potential biases introduced by the estimation process, which is a novel contribution for high-dimensional nonparametric models. To prove the validity of their confidence bands, they employ a Gaussian multiplier bootstrap procedure, a sophisticated technique for approximating the distribution of complex statistical processes.

Datasets

The paper evaluates its method on:

  • Synthetic Data: Generated from both sparse additive models (SpAM) and the more general ATLAS model (introduced by the authors).
  • Real-World Neuroscience Data:
    • Genomic Data: Analyzing the relationship between gene expression levels and riboflavin production.
  • Neural Imaging Data (ADHD-200 dataset): Investigating the connection between ADHD index and fMRI data from brain voxels.

Key Insights and Ideas

  • Bridging the Gap: The kernel-sieve hybrid approach successfully combines the local estimation power of kernel methods with the global approximation capabilities of splines, offering a more comprehensive estimation strategy.
  • Uncertainty Quantification for SpAMs: The paper provides a principled way to construct confidence bands for individual components of sparse additive models, which was a significant open problem, especially in high dimensions.
  • ATLAS Model: The introduction of the ATLAS (Additive Local Approximation with Sparsity) model generalizes SpAMs by allowing for more complex interactions and varying sparsity patterns, making the proposed methods applicable to a broader class of problems.
  • De-biasing is Crucial: The development of a de-biasing procedure is highlighted as a key technical innovation for achieving honest confidence bands in this challenging setting.

Key Results

  • The proposed kernel-sieve hybrid estimator is shown to be asymptotically honest, meaning the confidence bands it produces have the correct coverage probability in the limit.
  • Theoretical results establish the rate of convergence for the estimator, demonstrating its statistical efficiency.
  • Numerical experiments on synthetic and real data (genomic and fMRI) show that the method performs well in terms of coverage probability and area coverage for confidence bands, outperforming standard SpAM approaches in certain scenarios.
  • The application to real-world data demonstrates the method's ability to identify significant relationships and provide meaningful uncertainty estimates.

Limitations and Future Work

  • Assumption on Density: The paper relies on a stronger assumption on the density function of covariates (Assumption A1) compared to some previous work, which is necessary for kernel-type methods.
  • Generalization: While the ATLAS model is more general than SpAM, the paper notes that further generalizations might be possible.
  • Computational Complexity: Although an accelerated algorithm is proposed, the computational complexity can still be a challenge in very high-dimensional settings.
  • Future Work: The authors suggest that their methods can be extended to other classes of nonparametric models and that the theoretical framework can be adapted for more complex scenarios. The paper also hints at further applications and extensions of the ATLAS model.

Summary updated

Abstract

We develop a novel procedure for constructing confidence bands for components of a sparse additive model. Our procedure is based on a new kernel-sieve hybrid estimator that combines two most popular nonparametric estimation methods in the literature, the kernel regression and the spline method, and is of interest in its own right. Existing methods for fitting sparse additive model are primarily based on sieve estimators, while the literature on confidence bands for nonparametric models are primarily based upon kernel or local polynomial estimators. Our kernel-sieve hybrid estimator combines the best of both worlds and allows us to provide a simple procedure for constructing confidence bands in high-dimensional sparse additive models. We prove that the confidence bands are asymptotically honest by studying approximation with a Gaussian process. Thorough numerical results on both synthetic data and real-world neuroscience data are provided to demonstrate the efficacy of the theory. Supplementary materials for this article are available online.
BibTeX
@article{lu2019kernel,
  author = {Junwei Lu and Mladen Kolar and Han Liu},
  title = {{Kernel Meets Sieve: Post-Regularization Confidence Bands for Sparse Additive Model}},
  journal = {Journal of the American Statistical Association},
  year = {2019},
  doi = {10.1080/01621459.2019.1689984},
}
Figure