In scientific computing and machine learning, we often face a trade-off between cost and accuracy. To optimize a complex system—like a car's aerodynamics or a deep neural network—we can either run a few "high-fidelity" experiments (expensive, slow, but accurate) or many "low-fidelity" ones (cheap, fast, but noisy).
Traditionally, researchers treated these fidelities as a handful of discrete buckets: a "coarse" mesh vs. a "fine" mesh, or a 10% data subset vs. the full dataset. However, a new wave of research suggests that treating fidelity as a continuous coordinate—a sliding scale rather than a set of steps—unlocks significantly more efficient optimization and more accurate surrogates.
The Fidelity Coordinate
The core intuition behind this shift is that the parameters governing fidelity, such as mesh density in fluid dynamics or the number of training iterations in deep learning, are naturally continuous. When we force these into discrete categories, we lose the ability to understand how the system's behavior evolves as we invest more resources.
Kirthevasan Kandasamy and colleagues introduced the BOCA (Bayesian Optimisation with Continuous Approximations) framework to exploit this. In Multi-fidelity Bayesian Optimisation with Continuous Approximations, they argue that modeling the relationship between fidelity and performance using Gaussian Processes allows an optimizer to navigate the cost-accuracy trade-off dynamically. Instead of being stuck with "Level 1" or "Level 2," the algorithm can choose any arbitrary point on the spectrum to maximize information gain per unit of cost.
This approach is particularly resilient to "bad" low-fidelity data. If a low-fidelity approximation is poorly correlated with the truth, a continuous model can detect this trend and automatically shift its focus toward higher-fidelity evaluations, preventing the optimization from stalling on misleading, cheap data.
Modeling Accuracy as a Dynamical System
If fidelity is a continuous coordinate, how do we model the "path" from a low-fidelity guess to a high-fidelity truth? Recent work reimagines this transition as a continuous evolution governed by differential equations.
In Infinite-Fidelity Coregionalization for Physical Simulation, Shibo Li and Shandian Zhe use Neural Ordinary Differential Equations (ODEs) to represent physical outputs. By treating simulation fidelity as a coordinate, the model "learns the path" that the data takes as the mesh becomes finer. This allows the model to not only interpolate between known mesh sizes but also to extrapolate, predicting results at accuracies higher than any seen during training.
Similarly, WEI W. XING and Yuxin Wang proposed ContinuAR in ContinuAR: Continuous Autoregression For Infinite-Fidelity Fusion. By reformulating classic autoregressive models as linear differential equations, they achieved a 62,500x speedup in training time compared to previous state-of-the-art methods. This "infinite-fidelity fusion" treats the transition between precision levels as a smooth flow, allowing for the integration of data from any mesh density or solver tolerance without requiring fixed, nested structures.
Granular Resource Allocation
The practical benefit of this continuous view is most visible in hyperparameter tuning and large-scale simulations. When tuning a model, we often have multiple resource dimensions—like the number of training iterations and the size of the training dataset.
Jian Wu and Andrew Gordon Wilson demonstrated in Practical Multi-fidelity Bayesian Optimization for Hyperparameter Tuning that treating these as continuous variables allows for much faster convergence. Their trace-aware knowledge-gradient (taKG) method uses intermediate validation scores (the "trace" of a training run) to update its beliefs across the continuous fidelity space, finding optimal configurations faster than discrete methods like Hyperband.
This granular control is also becoming essential for developing scaling laws. As Paul Setinek and Johannes Brandstetter point out in Towards Multi-Fidelity Scaling Laws of Neural Surrogates in CFD, current scaling laws often treat data fidelity as a binary variable. Moving toward a continuous spectrum of mesh densities and time steps is the next frontier for predicting how scientific AI models will perform as they scale.
By treating fidelity as a continuous dimension, these researchers have moved beyond simple "shortcuts" and toward a unified framework where every data point, regardless of its cost, contributes to a single, evolving understanding of the target function.