Bayesian Optimization (BO) is the standard tool for optimizing expensive "black-box" functions, such as tuning hyperparameters or designing new materials. Traditionally, BO relies on Gaussian Processes (GPs) to act as a surrogate model. GPs provide elegant, closed-form uncertainty estimates that tell the optimizer where to explore next.
However, GPs suffer from a fundamental "cubic bottleneck": their computational cost grows as $O(N^3)$ with the number of observations $N$. This makes them impractical for problems involving thousands of data points or high-dimensional input spaces. To overcome this, researchers are increasingly replacing GPs with neural-based surrogates that offer linear scaling and better flexibility in complex environments.
The Cubic Bottleneck and the Neural Alternative
The primary limitation of Gaussian Processes is the need to invert a large covariance matrix. As Jasper Snoek and colleagues observed in Scalable Bayesian Optimization Using Deep Neural Networks, this $O(N^3)$ scaling prevents BO from being used in high-throughput settings. Their solution, DNGO, uses a Deep Neural Network to extract features and performs Bayesian linear regression only on the final layer. This shift reduces complexity to $O(N)$, allowing BO to scale to thousands of observations.
Neural surrogates offer three distinct advantages over traditional kernels:
- Linear Scaling: While GPs struggle beyond a few hundred points, neural architectures like those used in PHOENICS by Florian Häse scale linearly, enabling optimization over massive datasets.
- High-Dimensional Expressivity: Standard GPs often fail in spaces with more than 10 dimensions because they rely on simple Euclidean distance. Huong Ha and Santu Rana demonstrated that Bayesian Neural Networks (BNNs) consistently outperform GPs in high-dimensional Level Set Estimation by learning complex, non-linear representations without fixed kernels High Dimensional Level Set Estimation with Bayesian Neural Network.
- Handling Structured Data: Neural networks can process high-dimensional natural language embeddings or images, which are difficult for standard kernel-based methods to ingest Human-LLM Collaborative Feature Engineering for Tabular Data.
Quantifying Uncertainty Without Kernels
The core challenge of using neural networks for BO is Uncertainty Quantification (UQ). A standard neural network is "overconfident"—it provides a single prediction without knowing what it doesn't know. To guide exploration, the surrogate must provide epistemic uncertainty.
Researchers have developed several mechanisms to extract this "Bayesian" behavior from neural networks:
- Monte Carlo Dropout: By keeping dropout active during inference, the model produces a distribution of outputs that serves as a scalable proxy for GP uncertainty Large-Batch, Neural Multi-Objective Bayesian Optimization.
- Randomized Prior Networks (RPNs): Mohamed Aziz Bhouri and Paris Perdikaris utilize ensembles of neural networks with fixed random priors. This approach allows BO to handle output spaces exceeding 65,000 dimensions, such as dense sensor grids, which would be computationally impossible for GPs Scalable Bayesian optimization with high-dimensional outputs using randomized prior networks.
- Hamiltonian Monte Carlo (HMC): For more robust uncertainty, Jost Tobias Springenberg and Frank Hutter introduced BOHAMIANN, which uses a scale-adapted SGHMC to produce well-calibrated posterior estimates in high-dimensional search spaces Bayesian Optimization with Robust Bayesian Neural Networks.
From Sample Efficiency to Iteration Efficiency
In many modern engineering tasks, the bottleneck is no longer just the number of samples, but the time it takes to compute the next experiment. In Large-Batch, Neural Multi-Objective Bayesian Optimization, Vahid Babaei argues for a shift toward iteration efficiency. By using BNNs, they can select batches of up to 20,000 points simultaneously. This massive parallelism is enabled by the fact that neural network inference time remains relatively constant even as system size increases, unlike traditional numerical methods Accelerating Optimal Experimental Design for Robust Synchronization of Uncertain Kuramoto Oscillator Model Using Machine Learning.
Specialized Architectures for Transfer and Physics
The flexibility of neural surrogates allows them to be tailored to specific problem structures:
- Neural Processes (NPs): These models combine the probabilistic nature of GPs with the scalability of neural networks. Ankush Chakrabarty used Attentive Neural Processes to calibrate "digital twins" of buildings, handling high-frequency operational data that would choke a GP Attentive Neural Processes and Batch Bayesian Optimization for Scalable Calibration of Physics-Informed Digital Twins.
- Transfer Learning: Neural surrogates are naturally suited for meta-learning. Ying Wei and Peilin Zhao demonstrated that Transfer Neural Processes can leverage data from thousands of past experiments to "few-shot" the optimization of a new task, reaching optima up to 10x faster Meta-learning Hyperparameter Performance Prediction with Neural Processes.
By breaking the cubic scaling of Gaussian Processes, neural surrogates are expanding the reach of Bayesian Optimization from simple hyperparameter tuning to massive-scale scientific discovery, high-dimensional engineering design, and real-time robotic adaptation.