Accelerated Bayesian Optimization throughWeight-Prior Tuning

Alistair Shilton, Sunil Gupta, Santu Rana, Pratibha Vellanki, Laurence Park, Cheng Li, Svetha Venkatesh, Alessandra Sutti, David Rubin, Thomas Dorin, Alireza Vahid, Murray Height, Teo Slezak

Abstract: Bayesian optimization (BO) is a widely-used method for optimizing expensive (to evaluate) problems. At the core of most BO methods is the modeling of the objective function using a Gaussian Process (GP) whose covariance is selected from a set of standard covariance functions. From a weight-space view, this models the objective as a linear function in a feature space implied by the given covariance K, with an arbitrary Gaussian weight prior ${\bf w} \sim \mathcal{N} ({\bf 0}, {\bf I})$. In many practical applications there is data available that has a similar (covariance) structure to the objective, but which, having different form, cannot be used directly in standard transfer learning. In this paper we show how such auxiliary data may be used to construct a GP covariance corresponding to a more appropriate weight prior for the objective function. Building on this, we show that we may accelerate BO by modeling the objective function using this (learned) weight prior, which we demonstrate on both test functions and a practical application to short-polymer fibre manufacture.

Introduction

Bayesian Optimization (BO) is the gold standard for optimizing "expensive" black-box functions, such as hyper-parameter tuning for neural networks or material design. Traditionally, BO starts with a generic Gaussian Process (GP) prior, assuming all features are equally important ($w \sim \mathcal{N}(0, I)$). However, in many real-world scenarios, researchers possess "auxiliary data"—results from previous experiments or similar systems—that could potentially guide the current optimization.

Standard transfer learning attempts to use this data directly, but it often fails if the auxiliary function is not a near-perfect match for the target task, leading to "negative transfer." This paper introduces a novel approach: instead of transferring function values, we transfer the underlying covariance structure by learning a specific weight prior from auxiliary data.

Methodology: Weight-Prior Tuning

The core innovation lies in shifting from the traditional "function-space" view of GPs to a "weight-space" view.

The Weight-Prior Concept

The objective function is modeled as a linear combination of features in a high-dimensional space. While standard BO assumes all feature weights have equal variance, the proposed method learns a diagonal covariance matrix ($\Sigma_\tau$) for these weights from auxiliary data. This effectively tells the optimizer which "directions" in the search space are likely to be most impactful before the first evaluation begins.

The m-Kernel Trick

To integrate this learned importance into the GP framework, the authors utilize "m-kernels" (tensor kernels). They prove a theorem for constructing a new, "re-weighted" Mercer kernel ($K^A_2$) using weights derived from a Support Vector Machine (SVM) trained on the auxiliary dataset. This re-weighted kernel is then used in standard acquisition functions like Expected Improvement (EI) or Upper Confidence Bound (UCB).

Experimental Evaluation

Synthetic Benchmarks

The method was tested on six classic optimization functions (e.g., Ackley, Rastringin). To simulate a difficult transfer learning environment, the authors used flipped (negated) versions of these functions as auxiliary data.

Performance comparison on synthetic benchmark functions with flipped auxiliary data.

As shown in the results, the Tuned-Prior (TP) variants (TP-GP-UCB-BO and TP-EI-BO) reached the global optimum significantly faster than standard BO and existing transfer learning methods like envGP or diffGP. Because the TP method focuses on feature importance rather than function values, it remained robust even when the auxiliary data was the exact opposite of the target.

Real-World Application: Short-Polymer Fibre Manufacture

The researchers applied the method to Short-Polymer Fibre (SPF) manufacturing, aiming to find machine settings that produce a specific target fiber length. Auxiliary data was sourced from an older manufacturing device (Device A) to optimize a newer, distinct device (Device B).

Optimization results for Short-Polymer Fibre (SPF) manufacture and a diagram of the experimental setup.

In this practical application, the TP method again outperformed all competitors. Notably, standard transfer learning methods performed worse than basic BO, illustrating the danger of negative transfer when devices differ significantly. The TP approach successfully extracted the shared underlying structure to accelerate the search.

Discussion and Limitations

While highly effective, the method has specific requirements:

  • Data Quality: It requires enough auxiliary data to characterize the covariance structure. If the auxiliary set is too small, the learned weights may be uninformative.
  • Scale Sensitivity: The benefit is reduced if the objective function varies on a much smaller scale than the auxiliary data.
  • Computational Cost: Evaluating the re-weighted kernel scales at $O(|A|^2)$, where $|A|$ is the size of the auxiliary dataset. For very large datasets, sparsification techniques would be necessary.

Conclusion

By focusing on transferring feature importance through weight-prior tuning rather than direct function values, this approach provides a robust and mathematically principled way to accelerate Bayesian Optimization. It effectively bridges the gap between generic BO and rigid transfer learning, offering significant performance gains in both synthetic and complex real-world manufacturing tasks.

Summary updated

Abstract

Bayesian optimization (BO) is a widely-used method for optimizing expensive (to evaluate) problems. At the core of most BO methods is the modeling of the objective function using a Gaussian Process (GP) whose covariance is selected from a set of standard covariance functions. From a weight-space view, this models the objective as a linear function in a feature space implied by the given covariance K, with an arbitrary Gaussian weight prior ${\bf w} \sim \mathcal{N} ({\bf 0}, {\bf I})$. In many practical applications there is data available that has a similar (covariance) structure to the objective, but which, having different form, cannot be used directly in standard transfer learning. In this paper we show how such auxiliary data may be used to construct a GP covariance corresponding to a more appropriate weight prior for the objective function. Building on this, we show that we may accelerate BO by modeling the objective function using this (learned) weight prior, which we demonstrate on both test functions and a practical application to short-polymer fibre manufacture.
BibTeX
@misc{shilton2018accelerated,
  author = {Alistair Shilton and Sunil Gupta and Santu Rana and Pratibha Vellanki and Laurence Park and Cheng Li and Svetha Venkatesh and Alessandra Sutti and David Rubin and Thomas Dorin and Alireza Vahid and Murray Height and Teo Slezak},
  title = {{Accelerated Bayesian Optimization throughWeight-Prior Tuning}},
  year = {2018},
  eprint = {1805.07852},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1805.07852},
}
Figure