When optimizing expensive black-box functions—such as designing a machine tool bed, tuning high-performance computing workflows, or discovering genetic perturbations—evaluating a single design can take hours or days. To navigate this constraint, engineers and researchers rely on surrogate models (like Gaussian Processes or Deep Neural Networks) to act as computationally cheap stand-ins for the true system.
Traditionally, machine learning models are trained to achieve maximum global fidelity, minimizing average error across the entire parameter space. However, a growing body of research demonstrates that pursuing global accuracy is not only computationally wasteful but often counterproductive for optimization. Instead, the key to highly efficient optimization is localized surrogate refinement: intentionally biasing the model's accuracy toward the most promising regions of the search space.
The Flaw of Global Fidelity
Why is a globally accurate model suboptimal for finding an optimum? The answer lies in how model capacity is allocated.
Surrogate models have finite representational capacity. When forced to minimize error across an entire domain, they must allocate parameters and training data to fit low-performance valleys and flat plains that have no bearing on the final optimal solution. As Natalie Maus et al. demonstrate in Approximation-Aware Bayesian Optimization, wasting limited model capacity on these irrelevant regions directly degrades search efficiency.
Furthermore, attempting to fit a complex global landscape often forces models to smooth out sharp, high-performing peaks. Yunyue Wei et al. show in Scalable Bayesian Optimization via Focalized Sparse Gaussian Processes that standard sparse approximations over-smooth localized maxima when trying to maintain uniform fidelity. By shifting the focus to local precision, the surrogate can capture these sharp peaks without being dragged down by distant data points.
This issue compounds drastically in high-dimensional spaces where the volume of the search space grows exponentially. As Yuxuan Yin et al. argue in High-Dimensional Bayesian Optimization via Semi-Supervised Learning with Optimized Unlabeled Data Sampling, achieving uniform global accuracy becomes mathematically intractable with sparse data. Focusing instead on local generalization near the current best-performing points allows the optimizer to bypass the curse of dimensionality entirely.
Decision-Driven Calibration
Optimization is fundamentally an act of comparison, not absolute estimation. To find the best configuration, an algorithm does not need to know the exact value of a poor design; it only needs to know with confidence that it is poor.
As Carlos Ansótegui et al. point out in Model-Based Genetic Algorithms for Algorithm Configuration, effective configuration requires models that prioritize precision in high-performance regions, even if they are highly inaccurate when predicting poor outcomes. This concept is visualized below:
Global Fit (Suboptimal for Optimization) Local Fit (Optimum-Centric)
___ ___
/ \ / * \ <- High Precision
_____/ \_____ _____/ \_____
/ \ / \
( Error minimized ) ( High error allowed )
\__equally everywhere_/ \__in low-value zones_/
This decision-driven perspective is validated across several distinct domains:
- HPC Auto-Tuning: In multi-component workflows, Tong Shu et al. demonstrate in Bootstrapping in-situ workflow auto-tuning via combining performance models of component applications that auto-tuners drastically reduce their data budget by remaining highly inaccurate for poorly performing configurations while sharpening precision around the optimum.
- Continuous Treatments: In decision-making contexts, Bo Li et al. show in Counterfactual Prediction for Outcome-Oriented Treatments that maximizing local accuracy specifically at the peak of an outcome curve minimizes selection regret, even if the model's average error across the entire treatment range is high.
- Biological Discovery: In high-dimensional genetic spaces, Yanke Li et al. show in BioBO: Biology-informed Bayesian Optimization for Perturbation Design that focusing representational capacity on top-tier candidates allows the system to effectively distinguish between high-value targets during acquisition.
Adaptive Sampling Strategies
How do we construct a model that is highly accurate only where it matters? The most direct mechanism is to bias the data collection process itself through adaptive, neighborhood-focused sampling.
Instead of spreading evaluations uniformly, the optimization loop can iteratively place new high-fidelity samples exclusively within promising sub-regions. Early work by Ahmed Mohamed Abouelyazied Abdallh et al. in Two-level refined direct method for electromagnetic optimization and inverse problems proved that refining surrogates solely in regions predicted to contain the optimum allows convergence with a fraction of the computational budget required by global metamodels.
This adaptive sampling loop has been refined across multiple optimization paradigms:
- Topology Optimization: Changyu Deng et al. developed a framework in Self-directed online machine learning for topology optimization that restricts online neural network training to samples generated within the immediate vicinity of the current predicted optimum, enabling high-resolution local mapping.
- Chemical Process Synthesis: In Data augmentation driven by optimization for membrane separation process synthesis, Bernardetta Addis et al. demonstrate that densifying datasets specifically within the sub-regions explored by optimization algorithms yields high-fidelity designs while minimizing expensive simulation overhead.
- Multi-Objective Trade-offs: This principle extends naturally to multi-objective problems. Augustina C. Amakor et al. show in Surrogate-assisted multi-objective design of complex multibody systems that focusing refinement samples strictly on the predicted Pareto front yields highly accurate design trade-offs without the prohibitive cost of a globally perfect approximation.
Tailoring the Loss Landscape
Biasing the training data is highly effective, but it can be augmented by changing how the surrogate model evaluates its own errors. By modifying loss functions and calibration targets, researchers can force models to prioritize the "tails" of distributions where optimal solutions reside.
For example, Aurélien Pion et al. introduce a targeted calibration method in Goal-Oriented Lower-Tail Calibration of Gaussian Processes for Bayesian Optimization that specifically calibrates the lower tail of a Gaussian Process. By ensuring the model accurately predicts the distribution of the best possible values rather than average ones, they prevent the optimizer from being misled by errors in high-value, irrelevant regions.
Similarly, in evolutionary optimization, Noritaka Tsukamoto et al. demonstrate in Incorporation of Hypervolume Approximation with Scalarizing Functions into Indicator-Based Evolutionary Multiobjective Optimization Algorithms that concentrating weight vectors locally around specific candidate solutions sharpens the precision of immediate survival decisions.
Ultimately, the historical assumption that optimization surrogates must be globally accurate maps has been replaced by a more efficient reality: a surrogate only needs to act as a highly precise magnifying glass over the peak. As Masashi YAMANAKA et al. noted in Optimum Design of Machine Tools Bed Using Neural Networks, biasing training data toward high-performance configurations yields superior optimum predictions. Conversely, failing to localize surrogates—as highlighted by Zhihai Ren et al. in A bi-stage surrogate-assisted hybrid algorithm for expensive optimization problems—leaves algorithms with insufficient precision to pinpoint the exact global optimum.
Key Foundational Work
Two-level refined direct method for electromagnetic optimization and inverse problems (2009) — contributed 2 concept elements to this direction.
Optimum Design of Machine Tools Bed Using Neural Networks (2005) — contributed 1 concept elements to this direction.