Scaling High Dimensional Optimization via Neural Surrogates

18 concepts13 papers1 proposal

In engineering and the physical sciences, optimization often requires evaluating expensive "black-box" systems—whether simulating the aerodynamic drag of an airfoil, the chemical kinetics of a hypersonic reentry vehicle, or the behavior of an analog circuit. Because running these full-scale simulations can take hours or days, engineers traditionally rely on surrogate models. These models act as mathematical stand-ins, trained on a small set of expensive simulation data to quickly predict the performance of new designs.

For years, the gold standard for surrogate modeling has been the Gaussian Process (GP), the backbone of traditional Bayesian Optimization (BO). GPs are highly prized because they provide exact uncertainty quantification, telling an optimization algorithm not just what it predicts, but how confident it is.

However, traditional surrogates hit a hard mathematical wall when applied to modern engineering problems. As design spaces grow to thousands of variables, or when simulations output high-resolution spatial grids, traditional methods suffer from severe scalability bottlenecks. A rising paradigm addresses this by replacing classical statistical surrogates with deep neural networks and neural operators, fundamentally altering the computational complexity of high-dimensional design.

The Complexity Bottleneck of Traditional Surrogates

To understand why neural surrogates work, we must first understand why traditional Gaussian Processes fail in high dimensions.

The core limitation of a GP is its computational complexity. To compute the posterior distribution of a GP, one must construct and invert a covariance matrix of size $N \times N$, where $N$ is the number of collected data points. Matrix inversion scales cubically, $\mathcal{O}(N^3)$. As an optimization algorithm actively samples more points to find a global optimum, the computational overhead quickly becomes more expensive than the underlying simulation itself.

Furthermore, GPs struggle with the curse of dimensionality in the input and output spaces. As noted by Tran-The et al. in their development of NeuralBO: A black-box optimization algorithm using deep neural networks, traditional statistical methods break down when mapping highly complex, non-linear relationships across many parameters.

When the optimization problem moves into function spaces—where the inputs or outputs are continuous fields like fluid velocity or temperature grids—the problem worsens. Discretizing these fields into high-resolution grids causes the dimensionality to explode.

Deep neural networks (DNNs) circumvent the $\mathcal{O}(N^3)$ bottleneck entirely. The computational complexity of a neural network scales with the number of network parameters and the dataset size during training, and remains highly efficient during inference. In Black-box Optimization with Unknown Constraints via Overparameterized Deep Neural Networks, Trong et al. demonstrate that over-parameterized deep neural networks can approximate both unknown objective functions and complex constraints simultaneously, scaling far more favorably with dataset size than classical alternatives.

Linear Scaling in Infinite-Dimensional Spaces

The advantage of neural architectures becomes even more pronounced when moving from standard neural networks to neural operators, such as Fourier Neural Operators (FNOs). While standard neural networks map vectors to vectors, neural operators are designed to map functions to functions, making them uniquely suited for partial differential equations (PDEs).

Traditional numerical solvers or spectral methods scale poorly as grid resolution increases, often scaling quadratically or at $\mathcal{O}(N \log N)$. In contrast, neural operators exhibit a flat or linear scaling behavior relative to the resolution of the output coordinates.

Traditional Solvers / GPs:  O(N³) or O(N log N)  --> Explodes at high resolutions
Neural Operators:           O(N) or Constant     --> Scales smoothly to fine grids

This phenomenon is central to the framework proposed by Oliveira et al. in Thompson Sampling in Function Spaces via Neural Operators. By replacing traditional GPs with neural operators, the authors enable optimization to scale linearly with output coordinate resolution. This allows algorithms to optimize complex physical systems at resolutions that would be completely intractable for standard Bayesian optimization.

Similarly, in Scattering with Neural Operators, Mizera demonstrates that neural operators bypass iterative numerical solvers for physics simulations entirely. Once trained, the operator's inference time and memory requirements remain relatively flat even as the simulation grid refines, unlocking "zero-shot super-resolution" where a model trained on coarse data generalizes seamlessly to fine grids. This property is also leveraged by Lee et al. in opPINN: Physics-Informed Neural Network with operator learning to approximate solutions to the Fokker-Planck-Landau equation, where neural surrogates for complex integral operators maintain constant inference costs, outperforming traditional spectral methods as velocity modes increase.

Turning Black Boxes into Differentiable Landscapes

Beyond raw computational scaling, neural surrogates fundamentally change how we navigate a search space.

Traditional black-box optimization often relies on derivative-free methods like evolutionary algorithms or stochastic sampling. In a design space with thousands of variables, searching randomly or semi-randomly is incredibly inefficient.

Neural networks solve this because they are natively differentiable. When a neural network is trained to approximate a black-box objective, it creates a smooth, continuous mathematical proxy of that objective. Because the proxy is a neural network, we can compute exact gradients of the output with respect to the input variables via backpropagation.

In Neural Network Accelerated Implicit Filtering: Integrating Neural Network Surrogates With Provably Convergent Derivative Free Optimization Methods, Irwin shows that structured neural network surrogates allow optimization frameworks to perform gradient descent directly on the surrogate model itself. Instead of relying solely on stochastic sampling, the algorithm uses the surrogate's gradients to point directly toward promising regions, scaling efficiently to thousands of variables.

This gradient-based refinement is also the core mechanism behind Geodesic Convolutional Shape Optimization. In this work, Baqué et al. utilize geodesic convolutional neural networks to emulate fluid dynamics simulators. By mapping complex 3D shapes onto regular structures, they enable fast, GPU-accelerated, differentiable performance predictions that allow gradient-based shape refinement to easily outpace traditional industry standards.

Scalable Uncertainty Quantification

If we replace Gaussian Processes with neural networks, we face a new challenge: how do we maintain the exploration-exploitation balance required by Bayesian Optimization? A standard neural network only outputs a single point prediction, lacking the built-in uncertainty quantification of a GP.

To preserve Bayesian optimization without re-introducing cubic costs, modern architectures use specialized, lightweight neural ensembles.

Traditional GP:          Inverts full NxN covariance matrix (Accurate but Slow)
Randomized Prior (RPN):  Ensemble of networks with fixed random priors (Fast & Scalable)
Neural Operator + EpiNet: Single backbone + tiny epistemic network (Ultra-Lightweight)

In Scalable Bayesian optimization with randomized prior networks, Bhouri et al. utilize Randomized Prior Networks (RPNs) as surrogates. By training an ensemble of networks where each possesses a fixed, randomized prior, they achieve rigorous uncertainty quantification. This allows Bayesian optimization to scale to massive output spaces, such as high-resolution images or spatio-temporal grids, which are computationally prohibitive for standard GPs.

To optimize even more efficiently, the authors of Composite Bayesian Optimization in function spaces using NEON—Neural Epistemic Operator Networks (Guilhoto et al.) introduce a framework that pairs a neural operator backbone with a compact "EpiNet." This architecture isolates the uncertainty estimation into a tiny auxiliary network, providing state-of-the-art uncertainty quantification while using up to 40x fewer parameters than traditional deep ensembles, allowing BO to scale effectively to 3D design spaces and functional inputs.

Real-World Efficiency Gains

By breaking the computational bottlenecks of traditional statistical models and numerical solvers, neural surrogate modeling has unlocked massive speedups across diverse domains:

Ultimately, neural surrogate modeling works because it shifts the burden of high dimensionality from rigid, cubically scaling statistical formulas to flexible, linearly scaling parametric models. By leveraging the fast inference of neural operators, the differentiability of neural networks, and scalable uncertainty mechanics, this paradigm makes high-dimensional, infinite-dimensional, and continuous-field optimization practical for complex scientific and engineering workflows.

Key Foundational Work

Geodesic Convolutional Shape Optimization (2018) — contributed 1 concept elements to this direction.
Bayesian Optimization Approach for Analog Circuit Synthesis Using Neural Network (2019) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_high_dimensional_surrogate_modeling_3babfe47,
  title = {high-dimensional surrogate modeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/high-dimensional-surrogate-modeling/rdr_3babfe47299444109054adb86f0c7851},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_high_dimensional_surrogate_modeling_3babfe47_v1,
  title = {high-dimensional surrogate modeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/high-dimensional-surrogate-modeling/rdr_3babfe47299444109054adb86f0c7851/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake