Bayesian Optimization (BO) is the gold standard for finding the best settings for "black-box" functions—systems where we don't know the internal math and every experiment is expensive, such as tuning a 7-billion parameter language model or synthesizing a new chemical compound.
However, BO faces a fundamental "cold-start" problem. Before the algorithm can be "smart," it needs data to build a surrogate model (usually a Gaussian Process) of the landscape. Traditionally, this means starting with a handful of random guesses. In high-stakes environments, these random trials are a waste of time and money.
To solve this, researchers are moving away from "blind" starts toward informed initialization, leveraging historical data, human expertise, and Large Language Models (LLMs) to give the optimizer a "warm start."
The Cold Start Bottleneck
In standard BO, the surrogate model begins with a flat, uninformative prior. As Matthias Feurer and Frank Hutter note in Initializing Bayesian Hyperparameter Optimization via Meta-Learning, this lack of task-specific observations prevents the model from making informed predictions.
The optimizer is essentially blindfolded in a dark room. It must stumble around randomly until it hits enough walls to map the space. In domains like chemical research, where every physical experiment is costly, this initialization phase is often the most inefficient part of the entire process Dong Han et al., ChemBOMAS.
Transferring Experience from Past Tasks
The most established way to warm-start BO is Transfer Learning. If you have optimized a similar model on a different dataset before, you shouldn't have to start from scratch.
Martin Wistuba and colleagues in Learning Data Set Similarities for Hyperparameter Optimization Initializations proposed using "landmark features"—early, cheap performance results—to calculate how similar a new task is to previous ones. By picking starting points that worked well on similar historical tasks, the optimizer can skip the "stumbling" phase entirely.
More advanced methods, like those proposed by Ho Chung Leon Law and Dino Sejdinovic, use distributional transfer to create automated representations of datasets. This allows the system to perform "one-shot" hyperparameter predictions, reaching peak performance in just a few evaluations.
Foundation Models as Scientific Intuition
A recent and powerful shift in the field is the use of Large Language Models (LLMs) to provide zero-shot priors. Unlike traditional surrogate models, LLMs have "read" millions of papers and manuals. They possess a form of "scientific intuition" that mathematical models lack.
In LLAMBO, Tennison Liu and Mihaela van der Schaar demonstrate that LLMs can act as surrogate models themselves. By framing optimization as a natural language task, the LLM can suggest promising regions of the search space before a single experiment is even run.
Similarly, Zhuo Chen and Xinzhe Yuan introduced LABO, which treats the LLM as a low-cost "simulator." The system uses a gating mechanism to decide when to trust the LLM’s "gut feeling" and when to trigger a costly real-world experiment. This hybrid approach provides the speed of LLM reasoning with the statistical rigor of traditional BO.
Encoding Human and Domain Expertise
Sometimes the best prior isn't a database of past runs, but the "gut feeling" of an expert engineer. Standard BO often ignores this, treating the optimum as equally likely to be anywhere.
Anil Ramachandran and Sunil Gupta addressed this through space warping. By mathematically stretching regions the expert thinks are promising and shrinking "unlikely" areas, the optimizer spends its budget where it matters most.
To prevent "bad" expert advice from ruining the search, DynMeanBO uses a dynamic mean decay. The algorithm "listens" to the expert early on for a speed boost but gradually ignores the prior as more real-world data is collected, ensuring the search eventually converges to the true optimum regardless of the initial guess.
Model-Aware Initialization
Even without external data, we can be smarter about the first batch of samples. Standard BO uses "space-filling" designs (like Sobol sequences) that try to cover the whole map.
However, Carl Hvarfner and David Eriksson argue in HIPE that initialization should be "model-aware." Instead of just filling space, HIPE picks starting points that are specifically designed to help the surrogate model learn its own internal parameters (like lengthscales) as quickly as possible. This ensures that by the second or third batch, the model is already highly accurate.
Why It Works: The Intuition of Information Transfer
The core reason these methods work is that optimization tasks are rarely truly independent. * Meta-learning works because the physics of a machine or the landscape of a neural network often share structural similarities across datasets. * LLM-guidance works because the "latent knowledge" in foundation models acts as a high-dimensional map of human discovery. * Informed initialization works because it recognizes that the goal of the first few steps isn't just to find the optimum, but to "train the trainer"—to make the surrogate model useful as fast as possible.
By replacing the "blind" random start with these informed strategies, researchers have shown that we can reach optimal configurations up to 16x faster Jiale Lao, GPTuner, turning BO from a slow exploratory tool into a rapid, precision instrument for scientific discovery.