Bayesian Approximate Kernel Regression with Variable Selection

Lorin Crawford, Kris C. Wood, Xiang Zhou, Sayan Mukherjee

Abstract

Nonlinear kernel regression models are often used in statistics and machine learning because they are more accurate than linear models. Variable selection for kernel regression models is a challenge partly because, unlike the linear regression setting, there is no clear concept of an effect size for regression coefficients. In this paper, we propose a novel framework that provides an effect size analog of each explanatory variable for Bayesian kernel regression models when the kernel is shift-invariant—for example, the Gaussian kernel. We use function analytic properties of shift-invariant reproducing kernel Hilbert spaces (RKHS) to define a linear vector space that: (i) captures nonlinear structure, and (ii) can be projected onto the original explanatory variables. The projection onto the original explanatory variables serves as an analog of effect sizes. The specific function analytic property we use is that shift-invariant kernel functions can be approximated via random Fourier bases. Based on the random Fourier expansion we propose a computationally efficient class of Bayesian approximate kernel regression (BAKR) models for both nonlinear regression and binary classification for which one can compute an analog of effect sizes. We illustrate the utility of BAKR by examining two important problems in statistical genetics: genomic selection (i.e. phenotypic prediction) and association mapping (i.e. inference of significant variants or loci). State-of-the-art methods for genomic selection and association mapping are based on kernel regression and linear models, respectively. BAKR is the first method that is competitive in both settings.

Introduction: The Interpretability-Performance Trade-off

In statistical modeling, researchers often face a fundamental trade-off between interpretability and predictive power. Linear models serve as the gold standard for interpretability; they provide explicit "effect sizes" (coefficients) that quantify the contribution of each variable to the outcome. However, these models frequently fail to capture the complex, nonlinear interactions inherent in biological and physical systems.

Conversely, nonlinear kernel models, such as Support Vector Machines (SVMs) or Gaussian Processes, offer superior predictive accuracy by mapping data into high-dimensional spaces. The cost of this performance is the "black box" nature of the resulting models. In a kernel space, there is no direct correspondence between the model's parameters and the original input variables, making it nearly impossible to pinpoint which specific features drive the results. This limitation is particularly acute in fields like statistical genetics, where identifying the specific genes responsible for a trait is as important as predicting the trait itself.

The BAKR Framework

To address this gap, we introduce Bayesian Approximate Kernel Regression (BAKR). This framework brings the interpretability of linear regression to the high-performance world of nonlinear kernel models. The methodology rests on three primary pillars:

1. Random Fourier Features (RFF)

Utilizing Bochner’s Theorem, we recognize that any shift-invariant kernel (such as the Gaussian kernel) can be approximated by a linear combination of random cosine and sine functions. This approximation transforms a complex nonlinear problem into a "pseudo-linear" problem within a new feature space, allowing us to leverage the computational advantages of linear algebra.

2. The Effect Size Analog

We define a mathematical projection that maps the complex nonlinear function back onto the original input variables. This process yields a vector $\tilde{eta}$—an "effect size analog"—that represents the contribution of each variable, including its nonlinear interactions. This allows for variable selection in high-dimensional datasets where $p$ (the number of variables) often far exceeds $n$ (the number of samples).

3. Computational Efficiency via Factor Modeling

To ensure the model scales to large datasets, we employ a Bayesian empirical factor model based on the spectral decomposition of the approximate kernel. By utilizing eigenvectors and eigenvalues, we reduce the model's complexity, enabling efficient sampling via Markov Chain Monte Carlo (MCMC) methods.

Experimental Evaluation

We validated BAKR using a combination of simulated genetic architectures and massive real-world datasets:

  • Stock Mouse Dataset: Analyzing 129 physical and behavioral traits across ~2,000 mice with ~10,000 genetic markers. BAKR emerged as the top-performing model for 93% of the phenotypes, consistently outperforming both standard linear models and exact SVMs.
  • Wellcome Trust Case Control Consortium (WTCCC): In this landmark human genetics study covering seven diseases (including Type 1 Diabetes and Crohn’s disease), BAKR identified 29 significant genomic regions. Notably, it discovered 12 new loci that were missed by the original study’s linear analysis, demonstrating its power to uncover hidden associations.

Discussion and Key Insights

Our findings suggest that in high-dimensional regimes ($p \gg n$), smooth nonlinear functions can be remarkably well-approximated by linear projections without significant loss of information. Furthermore, because the Gaussian kernel can be viewed as an infinite Taylor expansion, BAKR implicitly tests for all higher-order interactions between variables. This avoids the "combinatorial explosion" that typically hinders studies of epistasis (gene-gene interactions).

BAKR also demonstrates significant robustness to kernel bandwidth selection. As long as the model is not "undersmoothed" (which would lead to capturing excessive noise), it remains a powerful and reliable selector of variables.

Conclusion and Future Directions

BAKR represents a significant step forward in making nonlinear models interpretable for the scientific community. It is the first method to remain competitive in both genomic selection (prediction) and association mapping (inference).

Future work will focus on several key areas:

  1. Disentangling Effects: Developing methods to distinguish between a variable's linear contribution and its nonlinear interaction contribution.
  2. Scalability: Transitioning from MCMC to Variational Inference to scale the model to datasets with millions of samples, such as the UK Biobank.
  3. Kernel Expansion: Extending the framework beyond shift-invariant kernels to other families, such as the Matérn covariance functions used in spatial statistics.

Summary updated

Abstract

Nonlinear kernel regression models are often used in statistics and machine learning because they are more accurate than linear models. Variable selection for kernel regression models is a challenge partly because, unlike the linear regression setting, there is no clear concept of an effect size for regression coefficients. In this paper, we propose a novel framework that provides an effect size analog of each explanatory variable for Bayesian kernel regression models when the kernel is shift-invariant --- for example, the Gaussian kernel. We use function analytic properties of shift-invariant reproducing kernel Hilbert spaces (RKHS) to define a linear vector space that: (i) captures nonlinear structure, and (ii) can be projected onto the original explanatory variables. The projection onto the original explanatory variables serves as an analog of effect sizes. The specific function analytic property we use is that shift-invariant kernel functions can be approximated via random Fourier bases. Based on the random Fourier expansion we propose a computationally efficient class of Bayesian approximate kernel regression (BAKR) models for both nonlinear regression and binary classification for which one can compute an analog of effect sizes. We illustrate the utility of BAKR by examining two important problems in statistical genetics: genomic selection (i.e. phenotypic prediction) and association mapping (i.e. inference of significant variants or loci). State-of-the-art methods for genomic selection and association mapping are based on kernel regression and linear models, respectively. BAKR is the first method that is competitive in both settings.
BibTeX
@article{crawford2017bayesian,
  author = {Lorin Crawford and Kris C. Wood and Xiang Zhou and Sayan Mukherjee},
  title = {{Bayesian Approximate Kernel Regression with Variable Selection}},
  journal = {Journal of the American Statistical Association},
  year = {2017},
  doi = {10.1080/01621459.2017.1361830},
}
Figure