Bayesian Approximate Kernel Regression With Variable Selection

Lorin Crawford, Kris C. Wood, Xiang Zhou, Sayan Mukherjee

TL;DR

This paper introduces Bayesian Approximate Kernel Regression (BAKR), a novel framework that efficiently performs variable selection in nonlinear kernel regression models. By leveraging random Fourier features to approximate kernel functions, BAKR provides an "effect size analog" for each variable, enabling interpretable insights and competitive performance in both prediction and association mapping tasks, particularly in high-dimensional genetic data.

What Problem Does This Paper Address?

Linear models have well-established methods for variable selection, where regression coefficients directly indicate the "effect size" of each predictor. However, extending this to nonlinear kernel regression models, which often offer superior predictive accuracy, has been a significant challenge. The inherent complexity of kernel methods makes it difficult to define and compute meaningful effect sizes for individual variables, hindering interpretability and targeted analysis. This gap is particularly pronounced in fields like statistical genetics, where identifying specific genetic variants (variables) associated with complex traits is crucial.

Methods

The core innovation of BAKR lies in its use of random Fourier features to approximate shift-invariant kernel functions. This approximation allows the authors to bridge the gap between the infinite-dimensional Reproducing Kernel Hilbert Space (RKHS) and the original covariate space. Specifically:

  • Random Fourier Bases: Shift-invariant kernels (like the Gaussian kernel) can be approximated by a linear expansion of random Fourier bases. This allows for a computationally efficient representation of the kernel function.
  • Approximate Kernel Matrix: Using these random Fourier features, an approximate kernel matrix ($\tilde{K}$) is constructed. This matrix is then used in a generalized kernel model (GKM).
  • Factor Model Representation: The approximate kernel matrix is decomposed into an orthogonal matrix of eigenvectors and a diagonal matrix of eigenvalues. This factor model representation reduces the dimensionality from $n$ (number of samples) to $q$ (number of dominant eigenvectors), significantly speeding up computation.
  • Projection for Effect Size Analog: A key contribution is defining an "effect size analog" ($\beta$) by projecting the nonlinear response onto the original explanatory variables. This projection is achieved through a linear map derived from the factor model coefficients. This analog allows for variable selection in a manner analogous to linear models.
  • Bayesian Hierarchical Model: A Bayesian hierarchical model is specified for the factor model coefficients, incorporating priors that induce shrinkage and allow for interpretable posterior inference. A Gibbs sampler is used for posterior inference.

Datasets

The paper evaluates BAKR on:

  • Simulated Data: Two simulation scenarios are used to assess model performance in terms of variable selection power and predictive accuracy, mimicking genetic architectures with varying proportions of additive and interaction effects.
  • Stock Mouse Dataset: A dataset containing 129 quantitative traits from a heterogeneous stock mouse population is used to evaluate predictive performance for genomic selection. This dataset is noted for containing related samples, which can influence interaction effects.
  • WTCCC Data: Data from the Wellcome Trust Case Control Consortium (WTCCC) study, comprising seven common diseases and associated SNPs, is used for association mapping. This dataset is a standard benchmark for evaluating genetic association methods.

Key Insights and Ideas

  • Bridging Nonlinearity and Interpretability: The most significant insight is the successful creation of an "effect size analog" for nonlinear kernel models. This allows for the interpretability typically found in linear models to be brought to more powerful nonlinear approaches.
  • Efficiency through Approximation: The use of random Fourier features is not just a theoretical curiosity; it provides a practical pathway to computationally efficient kernel methods, especially for large datasets.
  • Unified Framework: BAKR offers a unifying framework that can handle both regression and classification tasks, and importantly, it can compete with state-of-the-art methods in both genomic selection (prediction) and association mapping.
  • Robustness to Kernel Smoothness: The method shows robustness to different choices of kernel bandwidth, suggesting practical utility across various data characteristics.

Key Results

  • Predictive Accuracy: BAKR consistently outperforms parametric models (Bayes Ridge, Lasso, LMM) and is competitive with or superior to Support Vector Machines (SVMs) in out-of-sample prediction tasks across both simulated and real datasets. For instance, in the stock mouse dataset, BAKR achieved the best performance for 93% of the phenotypes.
  • Association Mapping Power: BAKR demonstrates strong performance in identifying significant SNPs and genomic regions in the WTCCC dataset, discovering new loci not identified by previous analyses. It successfully identified 29 significantly associated genomic regions, including 12 novel loci.
  • Computational Efficiency: The approximate kernel approach, combined with the factor model, makes BAKR computationally feasible for large-scale problems, handling datasets with thousands of covariates and samples.
  • Approximation Error: Simulations show that the Monte Carlo error introduced by the approximate kernel is negligible and does not significantly impact performance compared to posterior estimation variability.

Limitations and Future Work

  • Direct Identification of Interaction Drivers: While BAKR identifies associated variables, it doesn't directly pinpoint which specific component (linear vs. nonlinear) drives the association or the detailed nature of interaction effects. The authors suggest using posterior samples to infer covariance structures or using BAKR as a screening tool for pairwise interactions.
  • Scalability to Extremely Large Sample Sizes: While BAKR scales well to millions of variables, scaling to hundreds of thousands or millions of samples would require algorithmic innovations.
  • Future Directions: The authors suggest extending BAKR to model multiple responses, incorporating correlations between them to increase power and identify pleiotropic effects. They also mention the possibility of matching shift-invariant kernels to specific genetic architectures and exploring kernels like the Matérn covariance function for spatial applications.

Summary updated

Abstract

Nonlinear kernel regression models are often used in statistics and machine learning because they are more accurate than linear models. Variable selection for kernel regression models is a challenge partly because, unlike the linear regression setting, there is no clear concept of an effect size for regression coefficients. In this paper, we propose a novel framework that provides an effect size analog for each explanatory variable in Bayesian kernel regression models when the kernel is shift-invariant - for example, the Gaussian kernel. We use function analytic properties of shift-invariant reproducing kernel Hilbert spaces (RKHS) to define a linear vector space that: (i) captures nonlinear structure, and (ii) can be projected onto the original explanatory variables. This projection onto the original explanatory variables serves as an analog of effect sizes. The specific function analytic property we use is that shift-invariant kernel functions can be approximated via random Fourier bases. Based on the random Fourier expansion, we propose a computationally efficient class of Bayesian approximate kernel regression (BAKR) models for both nonlinear regression and binary classification for which one can compute an analog of effect sizes. We illustrate the utility of BAKR by examining two important problems in statistical genetics: genomic selection (i.e. phenotypic prediction) and association mapping (i.e. inference of significant variants or loci). State-of-the-art methods for genomic selection and association mapping are based on kernel regression and linear models, respectively. BAKR is the first method that is competitive in both settings.
BibTeX
@article{crawford2017bayesian,
  author = {Lorin Crawford and Kris C. Wood and Xiang Zhou and Sayan Mukherjee},
  title = {{Bayesian Approximate Kernel Regression With Variable Selection}},
  journal = {Journal of the American Statistical Association},
  year = {2017},
  doi = {10.1080/01621459.2017.1361830},
}
Figure