Probabilistic Surrogate Modeling for Black Box Optimization

496 concepts395 papers2 proposals

In fields ranging from materials design to deep learning, we often encounter optimization problems where evaluating a single candidate solution is incredibly expensive. Whether it requires running a multi-hour fluid dynamics simulation, synthesizing a new compound in a chemistry lab, or training a massive language model, we cannot afford the thousands of evaluations demanded by traditional grid search or genetic algorithms. We might have a total budget of only a few dozen attempts.

To solve these "black-box" problems efficiently, we need a strategy that learns from every single evaluation. This is the core intuition behind probabilistic surrogate modeling. Instead of guessing blindly, we construct a statistical model—a surrogate—that stands in for the true, expensive function. Crucially, this surrogate does not just predict what the performance of an untested point will be; it quantifies how certain it is about that prediction.

The Engine: Quantifying Ignorance

Why must a surrogate model be probabilistic? If we use a standard deterministic model, like a vanilla neural network, it will output a single predicted value $\hat{y}$ for any input $x$. If the network predicts a high value in an unexplored region, we might waste a costly evaluation there, only to find out the prediction was wildly inaccurate because the model had no data in that neighborhood.

A probabilistic surrogate model solves this by outputting a probability distribution over the possible outcomes, typically characterized by a mean $\mu(x)$ and a variance $\sigma^2(x)$. The variance represents the model's epistemic uncertainty—its awareness of its own ignorance.

True Function:  ----------------------- (Unknown)
Surrogate Mean:  ......._._._...........
Uncertainty:    [   ]   ()   [   ]   ()  (High variance far from data)
Data Points:          X        X

The classic mathematical tool for this task is the Gaussian Process (GP). A GP assumes that any collection of points follows a joint multivariate Gaussian distribution, governed by a kernel function that defines how points correlate based on their distance. When we observe a data point, the model's uncertainty at that exact location collapses to zero (or the noise floor). As we move away from known data points, the uncertainty grows.

This framework is highly versatile. For instance, L. Mihaela Păun et al. use GP emulators in \<scp>Markov chain Monte Carlo\</scp>with Gaussian processes for fast parameter estimation and uncertainty quantification in a\<scp>1D\</scp>fluid‐dynamics model of the pulmonary circulation to bypass expensive cardiovascular simulations. Similarly, Brodie Lawson demonstrates in Slow Recovery of Excitability Increases Ventricular Fibrillation Risk as Identified by Emulation how GP emulation can rapidly map out complex electrophysiological risk spaces that would otherwise be computationally intractable.

The reliability of these uncertainty estimates hinges on the assumption that the underlying function behaves smoothly. As Hideitsu Hino et al. explain in Introduction to Active Learning and Application to Material Engineering, this smoothness assumption allows the model to interpolate safely, ensuring that uncertainty grows predictably as we venture further into unmeasured territory.

The Policy: Balancing Exploration and Exploitation

Once we have a surrogate model that provides both an expected performance $\mu(x)$ and an uncertainty $\sigma(x)$, how do we decide where to sample next? We use an acquisition function, which acts as a mathematical policy to navigate the exploration-exploitation trade-off.

Consider two unobserved points in our search space:

  • Point A has a high predicted mean $\mu(A)$ but low uncertainty $\sigma(A)$. It is a safe bet (Exploitation).
  • Point B has a low predicted mean $\mu(B)$ but massive uncertainty $\sigma(B)$. It is a wild card that could either be a total failure or a breakthrough discovery (Exploration).

Acquisition functions like Expected Improvement (EI) or Upper Confidence Bound (UCB) combine $\mu(x)$ and $\sigma(x)$ into a single metric. For example, UCB scores a point as:

$$\text{UCB}(x) = \mu(x) + \beta \sigma(x)$$

where $\beta$ is a tuning parameter. A high $\beta$ forces the system to explore highly uncertain regions, while a low $\beta$ focuses on exploiting known peaks.

This balancing act enables efficient optimization across vastly different domains. In cognitive science, Romy Lorenz et al. apply this in Dissociating frontoparietal brain networks with neuroadaptive Bayesian optimization to dynamically navigate a high-dimensional space of cognitive tasks, finding the exact tasks that best differentiate brain networks using real-time fMRI feedback. In hardware optimization, Menghao Li et al. deploy uncertainty-aware surrogates in AdaTune: Adaptive Tensor Program Compilation Made Efficient to quickly discover optimal compilation configurations for deep learning models without executing thousands of slow hardware benchmarks.

Expanding the Surrogate Toolbox

While Gaussian Processes are the mathematical gold standard for probabilistic modeling, they scale poorly to large datasets because fitting them requires inverting a matrix, an operation that scales cubically ($O(N^3)$) with the number of data points. Consequently, researchers have developed alternative architectures to generate mean and variance estimates.

Deep Learning and Language Models

Can deep neural networks act as probabilistic surrogates? Standard networks struggle with calibration, but recent approaches show they can be adapted. In a notable paradigm shift, Jamison Meindl et al. demonstrate in GPTOpt: Towards Efficient LLM-Based Black-Box Optimization that fine-tuned Large Language Models can be trained to generate structured text outputs containing both a proposed action and its predicted mean and standard deviation. This allows the LLM to function as a calibrated probabilistic surrogate, matching formal GPs on continuous optimization tasks.

Ensemble Methods

Another scalable alternative is to use an ensemble of deterministic models. By training multiple models (such as Radial Basis Functions or neural networks) on the same data with different initializations or hyperparameters, the variance between their predictions can serve as a proxy for uncertainty.

The authors of A bi-stage surrogate-assisted hybrid algorithm for expensive optimization problems, Zhihai Ren et al., utilize an ensemble of Radial Basis Functions with varying kernels to approximate expensive objective functions, using the diversity of the ensemble to guide the search. Similarly, Shadan Golestan et al. implement ensemble learning in Grey-Box Bayesian Optimization for Sensor Placement in Assisted Living Environments to quantify prediction reliability across a discrete spatial grid, reducing expensive physical evaluations by over 50%.

Handling Constraints and Multi-Fidelity Landscapes

In real-world engineering, we rarely optimize a single objective in isolation. We often face unmodeled constraints or have access to multiple data sources of varying quality and cost.

Constrained Optimization

Often, a candidate solution is only valid if it satisfies a hidden constraint. For example, a control system configuration might maximize speed but cause dangerous voltage spikes. Wenjie Xu et al. address this in CONFIG: Constrained Efficient Global Optimization for Closed-Loop Control System Optimization with Unmodeled Constraints by maintaining separate Gaussian Process surrogates for both the objective function and the unmodeled constraints. This allows the acquisition function to calculate the probability of feasibility alongside the expected improvement, ensuring the optimizer never samples in regions likely to violate safety thresholds.

Multi-Fidelity Optimization

In drug discovery or materials science, we often have access to cheap but inaccurate approximations (low-fidelity data, like rough simulations) alongside expensive, highly accurate measurements (high-fidelity data, like physical lab experiments).

High-Fidelity (Expensive):  o-----------o-----------o
                                \           \           \
Low-Fidelity (Cheap):      x---x---x---x---x---x---x---x---x

Instead of ignoring the cheap data, multi-fidelity surrogates learn the cross-correlation between fidelities. Peter Eckmann demonstrates this in MF-LAL: Drug Compound Generation Using Multi-Fidelity Latent Space Active Learning, using Stochastic Variational Gaussian Processes to model drug-binding affinities across multiple fidelity levels. The surrogate's uncertainty estimates dictate not just where to sample in the chemical space, but which fidelity is most cost-effective to query, maximizing the discovery rate under a strict computational budget.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 396 papers
  2. Version 1 · 2026-10-04 · 360 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_gaussian_process_surrogate_modeling_d4a825d2,
  title = {Gaussian process surrogate modeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/gaussian-process-surrogate-modeling/rdr_d4a825d2df3a4a84a056df66b0113703},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_gaussian_process_surrogate_modeling_d4a825d2_v2,
  title = {Gaussian process surrogate modeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/gaussian-process-surrogate-modeling/rdr_d4a825d2df3a4a84a056df66b0113703/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake