Kernel Pre-Training in Feature Space via m-Kernels

Alistair Shilton, Sunil Gupta, Santu Rana, Pratibha Vellanki, Cheng Li, Svetha Venkatesh, Laurence Park, Alessandra Sutti, David Rubin, Thomas Dorin, Alireza Vahid, Murray Height, Teo Slezak

Abstract

Bayesian optimization (BO) is a widely-used method for optimizing expensive (to evaluate) problems. At the core of most BO methods is the modeling of the objective function using a Gaussian Process (GP) whose covariance is selected from a set of standard covariance functions. From a weight-space view, this models the objective as a linear function in a feature space implied by the given covariance K, with an arbitrary Gaussian weight prior ${\bf w} \sim \mathcal{N} ({\bf 0}, {\bf I})$. In many practical applications there is data available that has a similar (covariance) structure to the objective, but which, having different form, cannot be used directly in standard transfer learning. In this paper we show how such auxiliary data may be used to construct a GP covariance corresponding to a more appropriate weight prior for the objective function. Building on this, we show that we may accelerate BO by modeling the objective function using this (learned) weight prior, which we demonstrate on both test functions and a practical application to short-polymer fibre manufacture.

Introduction

Bayesian Optimization (BO) is a powerful framework for optimizing "black-box" functions that are expensive to evaluate, such as hyper-parameter tuning for large-scale machine learning models or physical laboratory experiments. Typically, BO employs a Gaussian Process (GP) to model the objective function. However, standard GPs often rely on "off-the-shelf" kernels (like the Squared Exponential) that assume uninformative priors, often requiring numerous evaluations to learn appropriate feature scales.

In many real-world scenarios, auxiliary data from related experiments or simulations is available. Standard transfer learning often struggles when this auxiliary data has a different range or "shape" than the target task. This paper introduces a method to extract the underlying structural similarity—identifying which features are relevant and how they interact—from auxiliary data to jump-start the optimization of a new, expensive target function.

Methodology: Tuned-Prior BO

The proposed method, Tuned-Prior BO (TP-BO), bridges the gap between Support Vector Machines (SVMs) and Gaussian Processes using a weight-space perspective. A GP can be viewed as a linear function in a high-dimensional feature space with a Gaussian weight prior $w \sim \mathcal{N}(0, I)$. By utilizing "free kernels"—a family where the feature map is independent of the kernel order ($m$)—the authors leverage m-kernel theory to manipulate this weight prior.

The TP-BO algorithm follows three primary steps:

  1. Feature Extraction: An SVM is trained on an auxiliary dataset $A$ to identify important features.
  2. Kernel Transformation: The resulting SVM weights are used to construct a new, re-weighted kernel ($K^A_2$) via a specific transformation.
  3. Optimized BO: This "tuned" kernel serves as the covariance function for a GP in a standard BO loop.

This approach allows auxiliary data to "teach" the GP which directions in the search space are likely to be relevant before the first evaluation of the target function occurs.

Experimental Results

The researchers evaluated TP-BO on synthetic benchmarks and a real-world engineering application.

Synthetic Benchmarks

Six classic optimization test functions (e.g., Himmelblau, Ackley, Rastrigin) were used. To simulate challenging transfer learning conditions, "flipped" (negated) versions of these functions served as auxiliary data.

Convergence of BO methods on synthetic benchmark functions with flipped auxiliary sources.

As shown in Figure 1, the proposed TP-GPUCB-BO method consistently reached global optima faster than standard BO and outperformed existing transfer learning methods like Domain-Transfer Multi-Kernel Learning (DT-MKL).

Short Polymer Fibre (SPF) Manufacture

The goal was to optimize machine settings (e.g., channel width, flow speeds) to produce fibers of a specific length. The target task involved a new device (Device B), while auxiliary data came from an older device (Device A) with different operating ranges.

Optimization performance and experimental setup for the Short Polymer Fibre (SPF) task.

In this practical application (Figure 2), the algorithm converged significantly faster than baselines. It successfully captured shared sensitivities to geometric and flow parameters while ignoring absolute value differences between the two devices.

Discussion and Conclusion

The key insight of this work is that transfer learning does not require identical source and target functions; a shared covariance structure is sufficient for significant acceleration. By using SVMs as feature extractors for GPs, the paper provides a principled framework to inject prior knowledge into the optimization process.

While effective, the method's computational complexity scales with $O(|A|^2)$, suggesting a need for sparsification with very large auxiliary sets. Additionally, the authors note that if auxiliary data is uninformative, the kernel may "vanish," requiring a fallback strategy. Overall, "pre-training" the kernel proves to be a robust way to compensate for data scarcity in expensive optimization tasks.

Summary updated

Abstract

Bayesian optimization (BO) is a widely-used method for optimizing expensive (to evaluate) problems. At the core of most BO methods is the modeling of the objective function using a Gaussian Process (GP) whose covariance is selected from a set of standard covariance functions. From a weight-space view, this models the objective as a linear function in a feature space implied by the given covariance K, with an arbitrary Gaussian weight prior ${\bf w} \sim \mathcal{N} ({\bf 0}, {\bf I})$. In many practical applications there is data available that has a similar (covariance) structure to the objective, but which, having different form, cannot be used directly in standard transfer learning. In this paper we show how such auxiliary data may be used to construct a GP covariance corresponding to a more appropriate weight prior for the objective function. Building on this, we show that we may accelerate BO by modeling the objective function using this (learned) weight prior, which we demonstrate on both test functions and a practical application to short-polymer fibre manufacture.
BibTeX
@misc{shilton2018kernel,
  author = {Alistair Shilton and Sunil Gupta and Santu Rana and Pratibha Vellanki and Cheng Li and Svetha Venkatesh and Laurence Park and Alessandra Sutti and David Rubin and Thomas Dorin and Alireza Vahid and Murray Height and Teo Slezak},
  title = {{Kernel Pre-Training in Feature Space via m-Kernels}},
  year = {2018},
  eprint = {1805.07852},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1805.07852},
}
Figure