Bayesian Optimization (BO) is the standard approach for optimizing "black-box" functions that are expensive to evaluate, such as tuning the hyperparameters of a deep neural network or finding the optimal configuration for a particle accelerator. However, standard BO suffers from a "cold-start" problem: it begins every new task with no knowledge, often wasting dozens of expensive trials on random exploration before it understands the search space.
Research into Transfer Learning for Bayesian Optimization (TLBO) aims to eliminate this inefficiency. By leveraging data from historical optimization tasks, these methods allow an optimizer to "warm-start" its search, often reaching optimal configurations 50% to 90% faster than searching from scratch, as surveyed by Tianyi Bai and colleagues in Transfer Learning for Bayesian Optimization: A Survey.
Warm-Starting via Historical Configurations
The most direct way to transfer knowledge is to seed the initial search with configurations that performed well on similar past problems. Instead of starting with random points, researchers like Matthias Feurer and Frank Hutter proposed MI-SMBO in Initializing Bayesian Hyperparameter Optimization via Meta-Learning.
This approach works by identifying "meta-features" of a dataset (like the number of samples or features) and finding the most similar historical datasets in a database. The optimizer then evaluates the best-performing configurations from those similar tasks first. This simple mechanism provides an immediate, high-quality baseline, allowing the surrogate model to focus its search on promising regions of the space from the very first iteration.
Shaping the Surrogate Model with Informed Priors
While warm-starting seeds the points we evaluate, a more sophisticated approach is to change the prior of the surrogate model itself. Standard BO typically uses a Gaussian Process (GP) with a zero or constant mean, reflecting total ignorance of the function's shape.
In Prior-mean-assisted Bayesian optimization application on FRIB Front-End tunning, Kilean Hwang and team demonstrated that a neural network trained on historical data can serve as a "prior mean" (pmBO). This neural network provides a rough "guess" of the performance landscape, which the GP then refines with new data. Similarly, Petru Tighineanu and Felix Berkenkamp introduced Hierarchical GPs in Transfer Learning with Gaussian Processes for Bayesian Optimization. These models propagate uncertainty from related tasks in a principled way, bridging the gap between fast heuristics and slow, high-accuracy joint models.
Defining Task Similarity
The success of transfer learning depends on the ability to recognize when a new task is "similar" to an old one. Researchers have explored several ways to define this similarity:
- Meta-features: Jungtaek Kim uses statistical properties of datasets to map them into a meta-feature space in Learning to Transfer Initializations for Bayesian Hyperparameter Optimization.
- Behavioral Fingerprints: In On Using Pattern Matching Algorithms in MapReduce Applications, Nikzad Babaii Rizvandi treats CPU utilization patterns over time as a unique "fingerprint" to match new jobs to known ones.
- Visual Similarity: For robotics, Amaury Depierre uses geometric feature vectors of objects to retrieve high-performing grasping parameters from memory in Developmental Bayesian Optimization of Black-Box with Visual Similarity-Based Transfer Learning.
Transfer Across Adjustments and Contexts
Transfer learning is not just for moving between different datasets; it is also vital for iterative development. Danny Stoll and Frank Hutter showed in Hyperparameter Transfer Across Developer Adjustments that when a developer makes a small change to an algorithm (like adding a layer), they can reuse the HPO data from the previous version to speed up the next run by up to 2.6x.
In dynamic environments, such as autonomous racing, Lukas P. Fröhlich and Melanie Zeilinger utilize Contextual Bayesian Optimization in Model Learning and Contextual Controller Tuning for Autonomous Racing. By sharing information across similar environmental states (like road friction or vehicle mass), the system can rapidly adapt its controller to new conditions without needing to relearn the physics of racing from scratch.
Why It Works: Amortizing the Cost of Search
The core intuition behind these methods is that optimization should be an amortized process. Rather than treating every problem as a unique island, we treat them as variations of a shared structure. As Yulia Rubanova explores in Amortized Bayesian Optimization over Discrete Spaces, retaining the weights of an optimization policy across rounds allows a system to "learn to solve" a class of problems. Whether it is Tianqi Chen optimizing tensor programs in Learning to Optimize Tensor Programs or Leo Ardon solving logistics challenges, the goal is the same: use the past to navigate the future more efficiently.