Covariance Function Pre-Training with m-Kernels for Accelerated Bayesian Optimisation

Alistair Shilton, Sunil Gupta, Santu Rana, Pratibha Vellanki, Cheng Li, Laurence Park, Svetha Venkatesh, Alessandra Sutti, David Rubin, Thomas Dorin, Alireza Vahid, Murray HeightarXiv 2018 · DOI · arXiv

Abstract

The paper presents a novel approach to direct covariance function learning for Bayesian optimisation, with particular emphasis on experimental design problems where an existing corpus of condensed knowledge is present. The method presented borrows techniques from reproducing kernel Banach space theory (specifically m-kernels) and leverages them to convert (or re-weight) existing covariance functions into new, problem-specific covariance functions. The key advantage of this approach is that rather than relying on the user to manually select (with some hyperparameter tuning and experimentation) an appropriate covariance function it constructs the covariance function to specifically match the problem at hand. The technique is demonstrated on two real-world problems - specifically alloy design and short-polymer fibre manufacturing - as well as a selected test function.

Introduction

Bayesian Optimization (BO) is a powerful framework for optimizing expensive black-box functions, such as those found in chemical synthesis or hardware tuning. However, the efficiency of BO is heavily reliant on the choice of the covariance function (kernel), which encodes the similarity between points in the search space. Traditionally, practitioners select a standard kernel (e.g., Squared Exponential) and tune its hyperparameters, often a "best guess" approach that ignores available domain knowledge.

This work introduces a method to "pre-train" kernels by mining existing "condensed knowledge"—such as patents, technical handbooks, or previous experimental data. By leveraging the theory of $m$-kernels, the authors develop a mathematically rigorous way to automatically re-weight feature importance from auxiliary data, significantly accelerating the search for optimal solutions in complex engineering tasks.

Methodology: $m$-Kernels and Kernel Re-weighting

The core innovation lies in bridging the gap between historical data and the BO kernel using $m$-kernels, a concept derived from reproducing kernel Banach space theory.

Free Kernels

The authors define a class of "free kernels" where the underlying feature map is independent of the number of arguments ($m$). This property allows for the manipulation of the kernel's internal representation of the data. Table 1 illustrates the relationship between these free-kernels and standard 2-kernels.

Table 1: Comparison of Free-kernels and their corresponding 2-kernels for standard kernel types.

The rmK Algorithm

The proposed rmK algorithm follows a three-step process:

  1. Auxiliary Data Processing: An auxiliary dataset (representing "condensed knowledge") is used to train a Support Vector Machine (SVM).
  2. Feature Weighting: The weights learned by the SVM identify which features in the high-dimensional space are most relevant to the target problem.
  3. Kernel Construction: These weights are used to construct a new, problem-specific covariance function ($K^\mathcal{E}_2$).

This approach allows the BO loop to start with a kernel that already "knows" the likely geometry of the optimal regions based on historical evidence, rather than starting with a generic "blank slate" kernel.

Experimental Evaluation

The method was evaluated across three distinct scenarios:

  • Short Polymer Fibers (SPF): Optimizing fiber length for a new manufacturing device ("Device B") using 163 input/output pairs from a similar predecessor ("Device A").
  • Aluminium Alloy Design: Accelerating the design of a new hybrid alloy with high yield strength using a dataset derived from 46 existing patents (covering 2000, 6000, and 7000 series alloys).
  • Simulated Experiment: A synthetic test using a function with rotational symmetry to verify if the kernel could learn structural properties from 100 random vectors before optimization began.

Key Insights

  • Automated Feature Engineering: The method replaces manual feature weighting by human experts with an automated SVM-based approach that "reads" auxiliary data.
  • Effective Knowledge Transfer: The results demonstrate that auxiliary data does not need to be a perfect match to be beneficial. In the SPF task, data from a physically similar but different device successfully informed the optimization.
  • Geometric Reshaping: Unlike standard hyperparameter tuning (e.g., length-scale), this method reshapes the entire geometry of the feature space to match the problem's underlying structure.

Results and Discussion

The rmK approach demonstrated significant advantages:

  • Faster Convergence: In the alloy design experiment, the pre-trained kernel reached high utility scores in fewer iterations than standard BO.
  • Superior Accuracy: In the SPF task, rmK outperformed both standard BO and mixture kernel models across multiple acquisition functions (GP-UCB and EI).
  • Symmetry Learning: The simulated experiment confirmed that the kernel could successfully incorporate structural properties like rotational symmetry, leading to more efficient surface searches.

Limitations and Future Work

While promising, the authors noted that the performance of the method can be sensitive to the choice of acquisition function; for instance, results were less clear with Expected Improvement (EI) compared to GP-UCB in some tests. Additionally, there is a computational overhead associated with training the SVM and using a more complex re-weighted kernel, though this is typically negligible compared to the cost of physical experiments. Finally, the quality and relevance of the auxiliary data are critical, as misleading data could bias the optimization in an unproductive direction.

Summary updated

Abstract

The paper presents a novel approach to direct covariance function learning for Bayesian optimisation, with particular emphasis on experimental design problems where an existing corpus of condensed knowledge is present. The method presented borrows techniques from reproducing kernel Banach space theory (specifically m-kernels) and leverages them to convert (or re-weight) existing covariance functions into new, problem-specific covariance functions. The key advantage of this approach is that rather than relying on the user to manually select (with some hyperparameter tuning and experimentation) an appropriate covariance function it constructs the covariance function to specifically match the problem at hand. The technique is demonstrated on two real-world problems - specifically alloy design and short-polymer fibre manufacturing - as well as a selected test function.
BibTeX
@misc{shilton2018covariance,
  author = {Alistair Shilton and Sunil Gupta and Santu Rana and Pratibha Vellanki and Cheng Li and Laurence Park and Svetha Venkatesh and Alessandra Sutti and David Rubin and Thomas Dorin and Alireza Vahid and Murray Height},
  title = {{Covariance Function Pre-Training with m-Kernels for Accelerated Bayesian Optimisation}},
  year = {2018},
  eprint = {1802.05370},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1802.05370},
}
Figure