Prioritizing Local Surrogate Accuracy in Black Box Optimization

22 concepts15 papers1 proposal

When optimizing expensive black-box functions—such as designing a machine tool bed, tuning high-performance computing workflows, or discovering genetic perturbations—evaluating a single design can take hours or days. To navigate this constraint, engineers and researchers rely on surrogate models (like Gaussian Processes or Deep Neural Networks) to act as computationally cheap stand-ins for the true system.

Traditionally, machine learning models are trained to achieve maximum global fidelity, minimizing average error across the entire parameter space. However, a growing body of research demonstrates that pursuing global accuracy is not only computationally wasteful but often counterproductive for optimization. Instead, the key to highly efficient optimization is localized surrogate refinement: intentionally biasing the model's accuracy toward the most promising regions of the search space.

The Flaw of Global Fidelity

Why is a globally accurate model suboptimal for finding an optimum? The answer lies in how model capacity is allocated.

Surrogate models have finite representational capacity. When forced to minimize error across an entire domain, they must allocate parameters and training data to fit low-performance valleys and flat plains that have no bearing on the final optimal solution. As Natalie Maus et al. demonstrate in Approximation-Aware Bayesian Optimization, wasting limited model capacity on these irrelevant regions directly degrades search efficiency.

Furthermore, attempting to fit a complex global landscape often forces models to smooth out sharp, high-performing peaks. Yunyue Wei et al. show in Scalable Bayesian Optimization via Focalized Sparse Gaussian Processes that standard sparse approximations over-smooth localized maxima when trying to maintain uniform fidelity. By shifting the focus to local precision, the surrogate can capture these sharp peaks without being dragged down by distant data points.

This issue compounds drastically in high-dimensional spaces where the volume of the search space grows exponentially. As Yuxuan Yin et al. argue in High-Dimensional Bayesian Optimization via Semi-Supervised Learning with Optimized Unlabeled Data Sampling, achieving uniform global accuracy becomes mathematically intractable with sparse data. Focusing instead on local generalization near the current best-performing points allows the optimizer to bypass the curse of dimensionality entirely.

Decision-Driven Calibration

Optimization is fundamentally an act of comparison, not absolute estimation. To find the best configuration, an algorithm does not need to know the exact value of a poor design; it only needs to know with confidence that it is poor.

As Carlos Ansótegui et al. point out in Model-Based Genetic Algorithms for Algorithm Configuration, effective configuration requires models that prioritize precision in high-performance regions, even if they are highly inaccurate when predicting poor outcomes. This concept is visualized below:

Global Fit (Suboptimal for Optimization)      Local Fit (Optimum-Centric)
          ___                                           ___
        /     \                                       /  *  \  <- High Precision
  _____/       \_____                           _____/       \_____
 /                   \                         /                   \
(  Error minimized    )                       (  High error allowed )
 \__equally everywhere_/                       \__in low-value zones_/

This decision-driven perspective is validated across several distinct domains:

Adaptive Sampling Strategies

How do we construct a model that is highly accurate only where it matters? The most direct mechanism is to bias the data collection process itself through adaptive, neighborhood-focused sampling.

Instead of spreading evaluations uniformly, the optimization loop can iteratively place new high-fidelity samples exclusively within promising sub-regions. Early work by Ahmed Mohamed Abouelyazied Abdallh et al. in Two-level refined direct method for electromagnetic optimization and inverse problems proved that refining surrogates solely in regions predicted to contain the optimum allows convergence with a fraction of the computational budget required by global metamodels.

This adaptive sampling loop has been refined across multiple optimization paradigms:

Tailoring the Loss Landscape

Biasing the training data is highly effective, but it can be augmented by changing how the surrogate model evaluates its own errors. By modifying loss functions and calibration targets, researchers can force models to prioritize the "tails" of distributions where optimal solutions reside.

For example, Aurélien Pion et al. introduce a targeted calibration method in Goal-Oriented Lower-Tail Calibration of Gaussian Processes for Bayesian Optimization that specifically calibrates the lower tail of a Gaussian Process. By ensuring the model accurately predicts the distribution of the best possible values rather than average ones, they prevent the optimizer from being misled by errors in high-value, irrelevant regions.

Similarly, in evolutionary optimization, Noritaka Tsukamoto et al. demonstrate in Incorporation of Hypervolume Approximation with Scalarizing Functions into Indicator-Based Evolutionary Multiobjective Optimization Algorithms that concentrating weight vectors locally around specific candidate solutions sharpens the precision of immediate survival decisions.

Ultimately, the historical assumption that optimization surrogates must be globally accurate maps has been replaced by a more efficient reality: a surrogate only needs to act as a highly precise magnifying glass over the peak. As Masashi YAMANAKA et al. noted in Optimum Design of Machine Tools Bed Using Neural Networks, biasing training data toward high-performance configurations yields superior optimum predictions. Conversely, failing to localize surrogates—as highlighted by Zhihai Ren et al. in A bi-stage surrogate-assisted hybrid algorithm for expensive optimization problems—leaves algorithms with insufficient precision to pinpoint the exact global optimum.

Key Foundational Work

Two-level refined direct method for electromagnetic optimization and inverse problems (2009) — contributed 2 concept elements to this direction.
Optimum Design of Machine Tools Bed Using Neural Networks (2005) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_adaptive_surrogate_refinement_7113bcc9,
  title = {adaptive surrogate refinement},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/adaptive-surrogate-refinement/rdr_7113bcc9a7d2432a9835dabaa43aee73},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_adaptive_surrogate_refinement_7113bcc9_v1,
  title = {adaptive surrogate refinement},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/adaptive-surrogate-refinement/rdr_7113bcc9a7d2432a9835dabaa43aee73/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake