Informed Initialization for Bayesian Optimization

375 concepts309 papers5 proposals

Bayesian Optimization (BO) is widely celebrated for its sample efficiency when optimizing expensive black-box functions, such as tuning deep learning hyperparameters or designing physical materials. Yet, standard BO suffers from a glaring paradox: before it can be smart, it must be completely blind.

This inefficiency is known as the cold-start problem. To build its first internal surrogate model (typically a Gaussian Process), an optimizer requires an initial set of data points. Lacking any prior information, traditional frameworks resort to model-agnostic, space-filling designs like random sampling or Sobol sequences. In resource-constrained settings where a single evaluation can take hours or cost thousands of dollars, spending the first 10–20% of an experimental budget on blind guesses is highly wasteful.

To eliminate this computational bottleneck, researchers have developed methods to warm-start the optimization process. By injecting structure before the very first evaluation, these techniques allow optimizers to bypass random exploration and immediately target high-performing regions.


The Root of the Cold Start: Hyperparameter Blindness

To understand why warm-starting works, we must first understand why a cold start fails. A Gaussian Process (GP) relies on hyperparameters—such as lengthscales and signal variance—to determine how smoothly the target landscape behaves and how quickly performance drops off between points.

When an optimizer starts from scratch, it has no data to fit these hyperparameters. As a result, the acquisition function (the mathematical guide deciding where to sample next) lacks a predictive landscape. It cannot reliably balance exploration and exploitation, frequently getting trapped in local optima or wasting evaluations on obviously suboptimal regions.

Modern approaches solve this by providing the optimizer with an initial "intuition," categorized into three primary mechanisms: cross-task meta-learning, model-aware initialization, and multi-fidelity bootstrapping.


The most intuitive way to avoid a cold start is to remember the past. If you have already optimized hyperparameters for 100 image classification datasets, tuning a model for the 101st dataset should not require starting from scratch.

Mapping Task Correlations

Swersky et al. formalized this concept through Multi-Task Bayesian Optimization. Instead of treating the new task in isolation, their framework uses a shared Gaussian Process to explicitly model the correlations between historical source tasks and the current target task. This allows the optimizer to transfer information across domains, narrowing down the search space before the target task even begins.

To establish the theoretical foundation of this acceleration, Shilton et al. introduced Diff-GP, an algorithm that explicitly models the mathematical difference between source and target functions as a distinct Gaussian Process. The authors provided the first formal proof that transferring knowledge in this manner strictly tightens "regret bounds," mathematically demonstrating why and how prior tasks accelerate convergence.

Transferring Rankings and Meta-Features

Raw performance scores (like absolute validation error) often vary wildly across different datasets, making direct regression difficult. To circumvent this, Bardenet et al. developed SCoT (Surrogate-based Collaborative Tuning). Instead of predicting exact scores, SCoT focuses on transferring the relative ranking of configurations across tasks, leveraging the insight that a good learning rate on one dataset is highly likely to rank well on a similar one.

Standard BO:  [Blind Random Sampling] ──> [Slow Model Fitting] ──> [Optimization]
Warm-Start:   [Historical Tasks] ──(Meta-Features)──> [Informed Initial Points] ──> [Immediate Optimization]

To automate this selection process, Feurer introduced MI-SMBO, a meta-learning framework that computes descriptive "meta-features" (such as dataset size, number of classes, and data skewness) for the target task. By comparing these meta-features to a historical database, the system identifies the most similar past experiments and directly seeds the new search with their best-performing configurations.

Safeguarding Against Misleading Priors

A core risk of meta-learning is "negative transfer"—when historical data is completely different from the new task, leading the optimizer astray. To address this, Dai designed provably robust meta-BO algorithms (RM-GP-UCB and RM-GP-TS). These algorithms are mathematically guaranteed to converge to the global optimum even if the historical tasks are entirely uncorrelated or actively misleading, providing a safety net for real-world deployment.


Mechanism 2: Model-Aware and Structural Priors

What happens if you do not have a database of historical tasks? You can still avoid a blind cold start by designing initialization strategies that are inherently model-aware or structurally biased by expert intuition.

Predictive Exploration

Instead of using traditional space-filling designs that ignore the underlying surrogate model, Hvarfner et al. proposed HIPE (Hyperparameter-Informed Predictive Exploration). HIPE selects initial sampling points by balancing two distinct goals: filling the geometric search space and aggressively forcing the surrogate model to learn its own internal hyperparameters. By choosing initial points that maximize information gain about the model's parameters from the very first batch, HIPE drastically reduces the number of startup samples required to make the surrogate predictive.

Space Warping

Another powerful approach is to distort the geometry of the search space itself to reflect human intuition. Ramachandran et al. introduced a framework that uses expert priors to "warp" the search space.

Unwarped Space:  [  Low Prob  |  Equal Exploration  |  Low Prob  ]
Warped Space:    [ Low ]<───────[ High Exploration ]───────>[ Low ]

By mathematically stretching regions where an expert expects the optimum to reside and shrinking unpromising boundary areas, the initial Gaussian Process kernel naturally concentrates its early exploration on high-probability zones. This structural bias accelerates discovery without sacrificing the global convergence guarantees of the underlying acquisition function.


Mechanism 3: Multi-Fidelity and Collaborative Bootstrapping

When evaluating the true objective function is prohibitively expensive, the cold-start phase can be bypassed by initializing the surrogate model with cheap, lower-fidelity approximations.

Leveraging Low-Fidelity Proxies

Instead of running every initial configuration to full completion, multi-fidelity frameworks use early-stopped learning curves or downsampled datasets as a proxy. Li et al. developed MFES-HB, an optimization framework that constructs a weighted ensemble combining biased, low-fidelity data with sparse, full-fidelity results. This approach mitigates early-stage data scarcity, allowing the optimizer to build a functional surrogate model at a fraction of the computational cost.

This multi-fidelity scaling is particularly vital in large-scale machine learning. For instance, Chen et al. demonstrated with ADMIRE-BayesOpt that transferring data-mixture insights from small, inexpensive proxy models can warm-start the optimization of massive 7-billion-parameter language models, yielding optimization speedups exceeding 500%.

Collaborative Ensembles

In cloud computing and distributed systems, this bootstrapping process can happen across different users. Scheinert et al. introduced Karasu, a collaborative framework that shares anonymized profiling data across independent cluster workloads. By constructing a similarity-based ensemble of Gaussian Processes from shared user history, Karasu allows new users to completely bypass expensive initial profiling runs, solving the cold-start problem through decentralized cooperation.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_cold_start_optimization_e3795743,
  title = {cold-start optimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/cold-start-optimization/rdr_e3795743fc2d4a64827b75a86d23c371},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_cold_start_optimization_e3795743_v1,
  title = {cold-start optimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/cold-start-optimization/rdr_e3795743fc2d4a64827b75a86d23c371/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake