Overcoming the Smoothness Bias in Surrogate Models

30 concepts29 papers1 proposal

Surrogate models, particularly Gaussian Processes (GPs), are the bedrock of modern Bayesian Optimization (BO) and uncertainty quantification. They allow us to model expensive black-box functions by making a fundamental mathematical assumption: spatial smoothness. Standard kernels, such as the Radial Basis Function (RBF) or Matérn kernels, assume that if two inputs are close in the search space, their outputs must be highly correlated.

While this smoothness assumption makes the mathematics elegant and tractable, it creates a severe blind spot. Real-world systems are frequently filled with sharp cliffs, sudden phase transitions, discrete steps, and erratic noise. When a model biased toward smoothness encounters a discontinuity, its predictions and uncertainty estimates break down.

A growing body of research focuses on identifying exactly where this smoothness bias fails and developing mathematical and algorithmic frameworks to overcome it.

The Cost of Forced Smoothness

When a standard Gaussian Process is forced to fit a discontinuous or highly jagged landscape, it fails in two distinct ways: oversmoothing and uncertainty misestimation.

Oversmoothing Physical and Behavioral Cliffs

In many domains, a minor change in input can cause a catastrophic or qualitative shift in output. For example, in materials science, a tiny temperature shift can trigger an abrupt phase transition (Arroyave et al. et al.). In game theory, a slight adjustment to a strategy can cause a player's payoff to drop off a cliff (Marchesi et al.).

When standard GPs encounter these phenomena, they attempt to draw a smooth curve across the step function. As Cornford et al. point out in the context of modeling atmospheric fronts, this forced continuity blurs out the sharp, physically real boundaries, leading to highly inaccurate local representations.

Broken Uncertainty Estimates on Staircases

The smoothness bias also distorts the model's understanding of its own error. Consider a discrete or integer-valued search space, such as optimizing the number of layers in a neural network. The true objective landscape is a "staircase" or piecewise constant function—it remains perfectly flat until it jumps to the next integer.

As Garrido-Merchán et al. demonstrate, when a continuous GP tries to fit these flat steps, it oscillates or slopes between data points. This modeling error corrupts the model's uncertainty estimates, causing the optimization algorithm to look for improvements in regions that are actually flat, or to miss critical transitions entirely. Similarly, if the underlying noise of a system is highly discontinuous, standard heteroscedastic GP models fail to accurately allocate experimental replications (Dai et al.).

True Landscape (Staircase/Cliff):   ______│'------
Smooth GP Fit (Oversmoothed):       ───∕'─────────

This failure is particularly dangerous in safety-critical applications. As Wachi et al. observe, if an autonomous agent relies on a smooth GP to map out safe versus dangerous zones, the model will oversmooth the transition near "knife-edge" hazards. This causes the agent to drastically underestimate risk, potentially steering it into a catastrophic failure zone.

Breaking the Gaussian Limit with Heavy Tails

To fix this at a foundational level, researchers have re-examined the stochastic processes that underpin these models.

Standard infinite-width Bayesian Neural Networks (BNNs) are governed by the Central Limit Theorem: when you pool an infinite number of layers with bounded-variance weights, the network's outputs converge to a Gaussian Process. Because GPs are inherently mean-square continuous, they are mathematically trapped in their smoothness bias.

To break this trap, Loria et al. and Loria et al. altered the weight priors of infinite-width BNNs, utilizing heavy-tailed distributions with unbounded variance.

By invoking the Generalized Central Limit Theorem, the authors proved that these networks converge not to a GP, but to $\alpha$-stable processes. Because these processes naturally possess stochastic kernels, they do not suffer from mean-square continuity constraints. Instead of drawing a smooth line across a cliff, an $\alpha$-stable process can naturally model sharp jump discontinuities and localized spiky behaviors without oversmoothing the surrounding data.

Rethinking Summaries: Variational Inducing Kernels

Another mathematical bottleneck occurs in sparse Gaussian Processes. To scale GPs to large datasets, algorithms typically use a small set of "inducing points" to act as a summary of the global latent function.

However, this point-based summary implicitly assumes that the underlying function behaves smoothly between those points. If the latent forces driving the data are highly erratic—such as white noise or non-smooth stochastic differential equations—inducing points fail to capture the necessary information.

To resolve this, Álvarez et al. introduced Variational Inducing Kernels (VIKs). Instead of summarizing the latent space using discrete points, VIKs summarize the space using functions (kernels). This variational framework allows the model to scale linearly while robustly capturing highly non-smooth, erratic latent forces that would completely baffle a traditional point-based sparse GP.

When modifying the underlying stochastic process is too computationally expensive or theoretically complex, how do we optimize functions on these hostile, jagged landscapes?

Local and Model-Free Fallbacks

Global GP models degrade rapidly when an objective function features sudden reward drops or sharp performance cliffs. In these scenarios, maintaining a smooth global model of the search space becomes a liability.

As Letham et al. demonstrate, local search methods or model-free evolutionary strategies often outperform global surrogate-based optimization on highly discontinuous landscapes precisely because they do not rely on a global smoothness assumption.

Rather than abandoning surrogates entirely, another approach is to build an algorithmic fail-safe. Acerbi et al. developed Bayesian Adaptive Direct Search (BADS), which pairs the local modeling power of Bayesian Optimization with a robust, mesh-based direct search.

When the objective landscape is smooth, BADS leverages the sample efficiency of the GP surrogate. The moment the landscape becomes extremely jagged or discontinuous—causing the surrogate's local approximations to fail—the algorithm seamlessly falls back on the mesh-based search to maintain convergence guarantees.

Non-Smooth Surrogates

Finally, practitioners can swap out Gaussian Processes for models that inherently tolerate discontinuities. For instance, in complex engineering design or materials discovery where systems exhibit sharp performance cliffs or phase transitions, standard polynomial basis functions and GPs fail (He et al.). In these settings, authors suggest utilizing alternative surrogates like additive regression trees, which naturally partition the search space into discrete steps and handle jagged response surfaces natively (Arroyave et al.).

Key Foundational Work

Efficient Multioutput Gaussian Processes through Variational Inducing Kernels (2010) — contributed 1 concept elements to this direction.
Adding Constrained Discontinuities to Gaussian Process Models of Wind Fields (1998) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_surrogate_model_limitations_bdb2c118,
  title = {surrogate model limitations},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/surrogate-model-limitations/rdr_bdb2c118d0384d83a47187a13bdbce02},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_surrogate_model_limitations_bdb2c118_v1,
  title = {surrogate model limitations},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/surrogate-model-limitations/rdr_bdb2c118d0384d83a47187a13bdbce02/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake