Transfer Learning for Accelerated Bayesian Optimization

This direction is merged into another direction: cold-start optimization. Its last version stays readable and citable here.

Version 1 of 134 papers42 conceptsfrozen 2026-10-04

Bayesian Optimization (BO) is the standard approach for optimizing "black-box" functions that are expensive to evaluate, such as tuning the hyperparameters of a deep neural network or finding the optimal configuration for a particle accelerator. However, standard BO suffers from a "cold-start" problem: it begins every new task with no knowledge, often wasting dozens of expensive trials on random exploration before it understands the search space.

Research into Transfer Learning for Bayesian Optimization (TLBO) aims to eliminate this inefficiency. By leveraging data from historical optimization tasks, these methods allow an optimizer to "warm-start" its search, often reaching optimal configurations 50% to 90% faster than searching from scratch, as surveyed by Tianyi Bai and colleagues in Transfer Learning for Bayesian Optimization: A Survey.

Warm-Starting via Historical Configurations

The most direct way to transfer knowledge is to seed the initial search with configurations that performed well on similar past problems. Instead of starting with random points, researchers like Matthias Feurer and Frank Hutter proposed MI-SMBO in Initializing Bayesian Hyperparameter Optimization via Meta-Learning.

This approach works by identifying "meta-features" of a dataset (like the number of samples or features) and finding the most similar historical datasets in a database. The optimizer then evaluates the best-performing configurations from those similar tasks first. This simple mechanism provides an immediate, high-quality baseline, allowing the surrogate model to focus its search on promising regions of the space from the very first iteration.

Shaping the Surrogate Model with Informed Priors

While warm-starting seeds the points we evaluate, a more sophisticated approach is to change the prior of the surrogate model itself. Standard BO typically uses a Gaussian Process (GP) with a zero or constant mean, reflecting total ignorance of the function's shape.

In Prior-mean-assisted Bayesian optimization application on FRIB Front-End tunning, Kilean Hwang and team demonstrated that a neural network trained on historical data can serve as a "prior mean" (pmBO). This neural network provides a rough "guess" of the performance landscape, which the GP then refines with new data. Similarly, Petru Tighineanu and Felix Berkenkamp introduced Hierarchical GPs in Transfer Learning with Gaussian Processes for Bayesian Optimization. These models propagate uncertainty from related tasks in a principled way, bridging the gap between fast heuristics and slow, high-accuracy joint models.

Defining Task Similarity

The success of transfer learning depends on the ability to recognize when a new task is "similar" to an old one. Researchers have explored several ways to define this similarity:

Transfer Across Adjustments and Contexts

Transfer learning is not just for moving between different datasets; it is also vital for iterative development. Danny Stoll and Frank Hutter showed in Hyperparameter Transfer Across Developer Adjustments that when a developer makes a small change to an algorithm (like adding a layer), they can reuse the HPO data from the previous version to speed up the next run by up to 2.6x.

In dynamic environments, such as autonomous racing, Lukas P. Fröhlich and Melanie Zeilinger utilize Contextual Bayesian Optimization in Model Learning and Contextual Controller Tuning for Autonomous Racing. By sharing information across similar environmental states (like road friction or vehicle mass), the system can rapidly adapt its controller to new conditions without needing to relearn the physics of racing from scratch.

The core intuition behind these methods is that optimization should be an amortized process. Rather than treating every problem as a unique island, we treat them as variations of a shared structure. As Yulia Rubanova explores in Amortized Bayesian Optimization over Discrete Spaces, retaining the weights of an optimization policy across rounds allows a system to "learn to solve" a class of problems. Whether it is Tianqi Chen optimizing tensor programs in Learning to Optimize Tensor Programs or Leo Ardon solving logistics challenges, the goal is the same: use the past to navigate the future more efficiently.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_transfer_learning_for_accelerated_bayesi_64571798,
  title = {Transfer Learning for Accelerated Bayesian Optimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/transfer-learning-for-accelerated-bayesian-optimization/rdr_64571798ccb94ac7b872a82b2a3d1a89},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_transfer_learning_for_accelerated_bayesi_64571798_v1,
  title = {Transfer Learning for Accelerated Bayesian Optimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/transfer-learning-for-accelerated-bayesian-optimization/rdr_64571798ccb94ac7b872a82b2a3d1a89/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}