Neural Surrogates for Scalable Bayesian Optimization

This direction is merged into another direction: Gaussian Process scalability, scalable surrogate modeling. Its last version stays readable and citable here.

Version 1 of 196 papers131 conceptsfrozen 2026-10-04

Bayesian Optimization (BO) is the standard tool for optimizing expensive "black-box" functions, such as tuning hyperparameters or designing new materials. Traditionally, BO relies on Gaussian Processes (GPs) to act as a surrogate model. GPs provide elegant, closed-form uncertainty estimates that tell the optimizer where to explore next.

However, GPs suffer from a fundamental "cubic bottleneck": their computational cost grows as $O(N^3)$ with the number of observations $N$. This makes them impractical for problems involving thousands of data points or high-dimensional input spaces. To overcome this, researchers are increasingly replacing GPs with neural-based surrogates that offer linear scaling and better flexibility in complex environments.

The Cubic Bottleneck and the Neural Alternative

The primary limitation of Gaussian Processes is the need to invert a large covariance matrix. As Jasper Snoek and colleagues observed in Scalable Bayesian Optimization Using Deep Neural Networks, this $O(N^3)$ scaling prevents BO from being used in high-throughput settings. Their solution, DNGO, uses a Deep Neural Network to extract features and performs Bayesian linear regression only on the final layer. This shift reduces complexity to $O(N)$, allowing BO to scale to thousands of observations.

Neural surrogates offer three distinct advantages over traditional kernels:

  1. Linear Scaling: While GPs struggle beyond a few hundred points, neural architectures like those used in PHOENICS by Florian Häse scale linearly, enabling optimization over massive datasets.
  2. High-Dimensional Expressivity: Standard GPs often fail in spaces with more than 10 dimensions because they rely on simple Euclidean distance. Huong Ha and Santu Rana demonstrated that Bayesian Neural Networks (BNNs) consistently outperform GPs in high-dimensional Level Set Estimation by learning complex, non-linear representations without fixed kernels High Dimensional Level Set Estimation with Bayesian Neural Network.
  3. Handling Structured Data: Neural networks can process high-dimensional natural language embeddings or images, which are difficult for standard kernel-based methods to ingest Human-LLM Collaborative Feature Engineering for Tabular Data.

Quantifying Uncertainty Without Kernels

The core challenge of using neural networks for BO is Uncertainty Quantification (UQ). A standard neural network is "overconfident"—it provides a single prediction without knowing what it doesn't know. To guide exploration, the surrogate must provide epistemic uncertainty.

Researchers have developed several mechanisms to extract this "Bayesian" behavior from neural networks:

From Sample Efficiency to Iteration Efficiency

In many modern engineering tasks, the bottleneck is no longer just the number of samples, but the time it takes to compute the next experiment. In Large-Batch, Neural Multi-Objective Bayesian Optimization, Vahid Babaei argues for a shift toward iteration efficiency. By using BNNs, they can select batches of up to 20,000 points simultaneously. This massive parallelism is enabled by the fact that neural network inference time remains relatively constant even as system size increases, unlike traditional numerical methods Accelerating Optimal Experimental Design for Robust Synchronization of Uncertain Kuramoto Oscillator Model Using Machine Learning.

Specialized Architectures for Transfer and Physics

The flexibility of neural surrogates allows them to be tailored to specific problem structures:

By breaking the cubic scaling of Gaussian Processes, neural surrogates are expanding the reach of Bayesian Optimization from simple hyperparameter tuning to massive-scale scientific discovery, high-dimensional engineering design, and real-time robotic adaptation.

Papers in this version

Showing 60 of 96 papers.

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_neural_surrogates_for_scalable_bayesian__b4b9d5c7,
  title = {Neural Surrogates for Scalable Bayesian Optimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/neural-surrogates-for-scalable-bayesian-optimization/rdr_b4b9d5c7cc584735964bbeab0ef97c2b},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_neural_surrogates_for_scalable_bayesian__b4b9d5c7_v1,
  title = {Neural Surrogates for Scalable Bayesian Optimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/neural-surrogates-for-scalable-bayesian-optimization/rdr_b4b9d5c7cc584735964bbeab0ef97c2b/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}