Continuous Multi-Fidelity Bayesian Optimization

18 concepts11 papers

In machine learning and physical simulation, we often face a fundamental trade-off: we can either run a fast, cheap experiment that provides a noisy approximation of the truth, or invest in a slow, expensive evaluation to get the "ground truth." Traditionally, researchers have managed this by selecting a few discrete levels of fidelity—such as training on 10% of a dataset or using a specific coarse mesh resolution.

However, many resource parameters are naturally continuous. The number of training iterations, the size of a data subset, and the density of a simulation mesh can all be adjusted with fine granularity. Recent research has shifted toward continuous multi-fidelity modeling, treating fidelity not as a set of discrete buckets, but as a continuous coordinate in the optimization space.

Fidelity as a Continuous Coordinate

Traditional multi-fidelity algorithms, like those discussed by Kandasamy et al. (2019), are often restricted to a finite set of approximations. This discretization forces a compromise: if the chosen low-fidelity level is too inaccurate, the optimizer may be misled; if it is too high, the cost savings vanish.

By treating fidelity as a continuous variable, we can model the objective function $f(x, z)$, where $x$ is the configuration (e.g., hyperparameters) and $z$ is the fidelity (e.g., dataset size). Kandasamy et al. (2017) introduced the BOCA framework, which uses Gaussian Processes to model this relationship. This approach allows the optimizer to dynamically navigate the cost-accuracy trade-off, selecting any arbitrary point within the fidelity range. A key advantage of this continuous view is resilience: if low-fidelity data is poorly correlated with the target, the system automatically shifts toward higher-fidelity evaluations to avoid stalling on misleading information.

Extrapolating Learning Curves

In hyperparameter optimization, the most common continuous fidelity dimensions are dataset size and training duration. Klein et al. (2017) developed FABOLAS, which treats dataset size as a continuous environmental variable. Instead of just correlating discrete tasks, the model learns a continuous function that describes how performance scales with data volume.

This allows the algorithm to "extrapolate" learning curves. By observing how a model performs on tiny, cheap subsamples, the surrogate model can predict performance on the full dataset without ever having evaluated that configuration at full scale. Similarly, Wu et al. (2019) proposed the trace-aware knowledge-gradient (taKG), which leverages the continuous "trace" of intermediate validation scores generated during training. By managing multiple continuous controls—like iterations and data size—simultaneously, these methods find optimal configurations significantly faster than discrete methods like Hyperband.

Fidelity as a Dynamical System

In physical simulations, such as Computational Fluid Dynamics (CFD), fidelity is often tied to mesh density. The transition from a coarse, low-resolution mesh to a fine, high-resolution mesh can be viewed as a continuous evolution of the physical output.

Li et al. (2022) introduced Infinite-Fidelity Coregionalization (IFC), which reimagines this transition as a dynamical system. By using Neural Ordinary Differential Equations (ODEs), the authors "learn the path" from low to high fidelity. This allows the model to:

  1. Interpolate between known mesh sizes.
  2. Extrapolate to higher accuracies (the "infinite-fidelity" limit) than those seen during training.

Xing et al. (2023) extended this concept with ContinuAR, which treats the transition between precision levels as a continuous evolution governed by linear differential equations. This approach avoids the need for fixed, nested data structures and provides massive speedups in training time compared to traditional multi-fidelity fusion.

The Frontier: Deep Surrogates and Scaling Laws

As we move toward more complex models, the limitations of standard Gaussian Processes become apparent. Li et al. (2020) and Li et al. (2021) have explored using Deep Neural Networks and Deep Auto-Regressive structures to capture non-linear correlations across continuous fidelity spaces. These deep surrogates can handle the complex, high-dimensional relationships found in modern machine learning architectures.

Finally, this perspective is beginning to influence how we think about scaling. Setinek et al. (2025) argue that current scaling laws—which typically treat data as a binary or discrete variable—must evolve. By modeling how performance scales across a continuous spectrum of simulation parameters like time steps and solver tolerances, researchers can more accurately predict the resources required to achieve a desired level of scientific accuracy.

Key Foundational Work

Multi-fidelity Bayesian Optimisation with Continuous Approximations (2017) — contributed 3 concept elements to this direction.
Practical Multi-fidelity Bayesian Optimization for Hyperparameter Tuning (2019) — contributed 2 concept elements to this direction.
Fast Bayesian hyperparameter optimization on large datasets (2017) — contributed 2 concept elements to this direction.
History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 10 papers
  2. Version 1 · 2026-10-04 · 21 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_continuous_fidelity_modeling_49f07c76,
  title = {continuous fidelity modeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/continuous-fidelity-modeling/rdr_49f07c76b69545bcae9027834c7ef357},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 2, frozen text):

@misc{lacuna_continuous_fidelity_modeling_49f07c76_v2,
  title = {continuous fidelity modeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/continuous-fidelity-modeling/rdr_49f07c76b69545bcae9027834c7ef357/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-19},
}

See something wrong? Report a mistake