Abstract
Bayesian optimization (BO) is a widely-used method for optimizing expensive-to-evaluate objective functions. At the core of most BO methods is the modeling of the objective function using a Gaussian Process (GP) whose covariance is selected from a set of standard covariance functions. From a weight-space view, this models the objective as a linear function in a feature space implied by the given covariance $K$, with an arbitrary Gaussian weight prior ${\bf w} \sim \mathcal{N}({\bf 0},{\bf I})$. In many practical applications, auxiliary data is available that shares a similar covariance structure with the objective but differs in form, making standard transfer learning difficult. This paper introduces a method to use such auxiliary data to construct a GP covariance corresponding to a more appropriate weight prior. We demonstrate that this "Tuned-Prior" approach significantly accelerates BO on both synthetic benchmarks and a practical manufacturing application.
Introduction
Bayesian Optimization is the standard approach for optimizing "black-box" functions where evaluations are costly, such as hyperparameter tuning or industrial process design. Most BO implementations rely on Gaussian Processes with generic kernels (e.g., Squared Exponential or Matérn) that assume all input features are equally relevant or rely on simple length-scale tuning.
However, in many engineering contexts, we possess auxiliary data from related tasks. While traditional transfer learning requires auxiliary data to be highly similar to the target function, this paper addresses a more flexible scenario: utilizing data that shares the same underlying structure (covariance) even if the actual values or scales are completely different. By baking learned feature importance into the kernel, we can significantly reduce the number of iterations required to find an optimum.
Tuned-Prior Bayesian Optimisation (TP-BO)
The proposed Tuned-Prior Bayesian Optimization (TP-BO) method follows a two-step process to inform the GP prior:
- Feature Importance Extraction: The auxiliary data ($A$) is used to identify relevant regions of the feature space. The authors train a Support Vector Machine (SVM) on this data using a specific class of "Free Kernels." The resulting SVM weights ($\alpha^A$) act as a proxy for feature importance, identifying which parts of the high-dimensional space define the problem's structure.
- Weight-Prior Tuning: Using a novel Mercer kernel construction ($K_2^A$), the SVM weights are integrated into the GP. In weight-space terms, this shifts the prior from a generic assumption ($w \sim \mathcal{N}(0, I)$) to an informed prior ($w \sim \mathcal{N}(0, \Sigma_\tau)$). This informed prior favors the features and directions found to be significant in the auxiliary data.
This tuned GP is then integrated into a standard BO loop, utilizing acquisition functions like Expected Improvement (EI) or Gaussian Process Upper Confidence Bound (GP-UCB) to navigate the search space of the target function $f$.
Experimental Evaluation
The researchers evaluated TP-BO against standard BO and existing transfer learning methods (like envGP) across two distinct domains:
- Synthetic Benchmarks: Six classic optimization functions (e.g., Himmelblau, Ackley, Rastringin). To test robustness, the auxiliary data consisted of 50 points drawn from the negated (flipped) version of the target function. This simulates a scenario where the auxiliary data provides structural information but misleading value information.
- Short-Polymer Fiber (SPF) Manufacture: A real-world design task aiming to optimize a manufacturing device ("Device B") to produce fibers of a specific target length. Auxiliary data was provided from 162 experiments conducted on an older, different hardware iteration ("Device A").
Results and Discussion
Synthetic Benchmarks
As illustrated in Figure 1, the TP-BO variants (represented by black lines) consistently outperformed standard methods. Despite the auxiliary data being "flipped" (negated), TP-BO reached the global minimum significantly faster. This highlights a key insight: the method prioritizes the structure of the landscape over the specific values. While traditional transfer learning methods like envGP were often misled by the negated values—performing worse than standard BO—TP-BO successfully leveraged the shared covariance structure to identify volatile and interesting regions of the search space.
Real-World Manufacturing
In the SPF manufacturing experiment (Figure 2), TP-BO reached the target fiber length with far fewer iterations than competing methods. In industrial settings where each evaluation involves physical hardware setup and material costs, this increased efficiency translates directly into significant time and resource savings. The bridge between discriminative learning (SVM) and generative modeling (GP) allows the system to effectively transfer knowledge across different physical devices.
Conclusion
TP-BO offers a robust framework for accelerating Bayesian Optimization by leveraging auxiliary data to tune the GP weight prior. By focusing on structural similarity rather than value similarity, the method remains effective even when auxiliary data is imperfect or misleading. Future work includes addressing computational scaling for very large auxiliary datasets and deriving formal regret bounds for this specific class of weight-prior tuning.
Summary updated