Most optimization algorithms treat every new problem as a blank slate. Whether tuning hyperparameters for a machine learning model, scheduling delivery trucks, or designing genetic experiments, standard solvers typically begin with random guesses or generic heuristics. This "cold start" approach is computationally expensive: in high-stakes domains like drug discovery or cloud resource allocation, the first few dozen "blind" trials can cost thousands of dollars or days of compute time.
A growing body of research suggests that optimization tasks are rarely unique. Instead, they belong to families of related problems that share underlying structural patterns. By transferring knowledge from historical tasks to new ones—a process known as "warm starting"—researchers are significantly reducing the search time required to find optimal solutions.
The Cost of Starting from Scratch
The fundamental inefficiency in traditional optimization is the lack of memory. As noted by Kartik Ahuja, Amit Dhurandhar, and Kush R. Varshney in Learning to Initialize Gradient Descent Using Gradient Descent, treating recurring tasks as independent events prevents solvers from leveraging shared structural properties.
In expensive domains, this inefficiency is a dealbreaker. For example, A. Cisse and colleagues describe the "cold start" in scientific Bayesian Optimization as a "wasteful expenditure of resources" where the search begins with random guesses in a space where every experiment is time-consuming (Language-Based Bayesian Optimization Research Assistant (BORA)).
Initializing Search with Historical Priors
The most direct way to bypass the cold start is to use historical data to bias the initial search distribution. Instead of starting with a wide, uniform search, the algorithm begins with a "warm" distribution centered on areas that worked for similar past problems.
Masahiro Nomura and Shuhei Watanabe demonstrated this with WS-CMA-ES, which warm-starts the Covariance Matrix Adaptation Evolution Strategy (Warm Starting CMA-ES for Hyperparameter Optimization). By mathematically defining task similarity, they initialize the search distribution based on prior results, making evolutionary strategies competitive even when the evaluation budget is extremely limited.
Similarly, in the realm of Large Language Model (LLM) evaluation, Jasper Dekoninck and Maximilian Baader developed Polyrating (Polyrating: A Cost-Effective and Bias-Aware Rating System for LLM Evaluation). By using Maximum A Posteriori (MAP) estimation to "recycle" data from previous benchmarks, they reduce the need for expensive human evaluations by up to 77%.
Encoding Problem Logic into Neural Solvers
While some methods transfer raw data points, others transfer the "logic" of the problem itself. This is often achieved by representing optimization problems as graphs.
In combinatorial optimization, Jian-Ya Ding and Le Song use Graph Convolutional Networks (GCNs) to predict the optimal values of variables in Mixed Integer Programs (MIPs) (Accelerating Primal Solution Findings for Mixed Integer Programs Based on Solution Prediction). By learning from a library of previously solved MIPs, the neural network provides a high-quality "primal solution" that gives the classical solver a massive head start, resulting in up to a 10x speedup.
This structural transfer extends to physical systems as well. Researchers have shown that Graph Neural Networks can learn the local interaction rules of energy landscapes from small systems and generalize them to much larger ones, providing a 100x speedup in finding ground states (Predicting ground state configuration of energy landscape ensemble using graph neural network).
Collaborative Optimization and Recommendation
A third perspective treats optimization as a recommendation problem. If a specific hyperparameter configuration performed well on ten different image classification datasets, it is likely a good starting point for the eleventh.
The SCoT framework by Rémi Bardenet and Michele Sebag uses this intuition to rank configurations across datasets (Collaborative hyperparameter tuning). Rather than predicting raw performance, they focus on the relative ranking of configurations, which is more stable across different tasks.
This "collaborative" approach is also seen in Probabilistic Matrix Factorization for Automated Machine Learning, which treats picking a machine learning pipeline like Netflix suggests movies. By factorizing a matrix of "pipelines vs. datasets," the system can infer which pipeline will work on a new dataset based on its similarity to historical ones.
Sharing Knowledge via Hierarchical Models
For tasks that occur concurrently or in rapid succession—such as managing hundreds of advertising campaigns—knowledge transfer can happen through shared Bayesian priors.
Runzhe Wan and Lin Ge introduced Multi-Task Thompson Sampling (MTTS), which uses Bayesian Hierarchical Models to share information across thousands of bandit tasks (Metadata-based Multi-Task Bandits with Bayesian Hierarchical Models). By using task-specific metadata (like user demographics), the algorithm provides a "warm start" for new tasks while remaining robust to individual differences, significantly reducing the "regret" (wasted opportunity) during the initial learning phase.
The Path Forward: Discovery Spaces and Agents
The frontier of this research involves creating persistent "memory" for optimization systems. Burkhard Ringlein and colleagues proposed Discovery Spaces, a data abstraction that decouples the optimization algorithm from the search space (Efficient and Reuseable Cloud Configuration Search Using Discovery Spaces). This allows different experiments to transparently reuse data, enabling a 90% speedup in cloud configuration tasks.
As optimization becomes more autonomous, agents like BioDiscoveryAgent (Yusuf H. Roohani et al., BioDiscoveryAgent: An AI Agent for Designing Genetic Perturbation Experiments) are beginning to combine these data-driven warm starts with human-like reasoning and biological priors, moving closer to a world where "starting from scratch" is a relic of the past.