The Fidelity Correlation Gap in MultiFidelity Optimization

44 concepts35 papers1 proposal

In machine learning and engineering design, we routinely face a frustrating bottleneck: evaluating our true objective function is painfully expensive. Whether it means training a deep neural network for weeks, running a high-resolution plasma micro-turbulence simulation, or conducting physical experiments with an inertial confinement fusion reactor, evaluating the "ground truth" costs significant time and capital.

To bypass this bottleneck, we use multi-fidelity optimization. Instead of relying solely on the expensive ground truth (high-fidelity), we leverage cheap approximations (low-fidelity)—such as training a model for only a few epochs, downsampling a dataset, or running a coarse numerical simulation.

The core promise of this paradigm is intuitive: use the cheap model to rapidly explore the search space, and use the expensive model only to verify and refine the best candidates. However, this entire machinery hinges on a single, often unexamined assumption: that the low-fidelity proxy actually correlates with the high-fidelity reality. When this correlation holds, multi-fidelity methods feel like magic. When it breaks, they can fail catastrophically.


Why It Works: Correlation, Not Accuracy

It is a common misconception that a low-fidelity surrogate model must be accurate to be useful. In reality, absolute accuracy is irrelevant; rank correlation is what drives multi-fidelity acceleration.

Imagine an autonomous vehicle simulator that consistently underestimates a robot's braking distance by exactly 5 meters. In terms of absolute accuracy, the simulator is poor. However, if it correctly predicts that Configuration A brakes faster than Configuration B, its rank correlation is perfect ($r = 1.0$).

As shown by Rachel Luo, simulators do not require high absolute accuracy to drastically reduce the statistical variance of real-world performance estimation, provided their outputs correlate with reality. In statistical terms, the low-fidelity model acts as a control variate. High correlation allows an algorithm to achieve the same level of statistical confidence using up to 80% fewer expensive physical tests.

This principle spans multiple domains:

  • Controller Testing: Shahrooei et al. demonstrate that simplified models with high sensor noise or reduced numerical precision can successfully guide Bayesian optimization toward safety violations, because they preserve the relative performance trends of the parameter space.
  • Plasma Physics: Konrad et al. combine data-driven sparse grids with multi-fidelity Monte Carlo sampling, showing that as long as the surrogate maintains high correlation with the ground truth, it can slash simulation times from eight days down to one hour without sacrificing confidence.

The Tipping Point: When Low-Fidelity Data Harms

If a little correlation is useful, is more low-fidelity data always better? Not necessarily. Incorporating a low-fidelity proxy introduces a trade-off between computational savings and statistical noise. If the proxy's correlation with the ground truth drops too low, it actively degrades the system.

High Correlation (r > Threshold)  --> Explores space cheaply, accelerates search
Low Correlation  (r < Threshold)  --> Injects noise, misleads optimizer, wastes budget

Zaytsev et al. mathematically formalized this boundary using a minimax approach to variable fidelity data interpolation. They proved the existence of a strict critical threshold for correlation. If the cross-fidelity correlation falls below this mathematical tipping point, attempting to blend the data sources introduces more error than it resolves. In such cases, the mathematically optimal strategy is to completely discard the low-fidelity data and allocate the entire computational budget exclusively to expensive, high-fidelity samples.


The Fidelity-Performance Correlation Gap

The most insidious failure mode of multi-fidelity systems occurs when the proxy correlates well with the ground truth on average, but fails locally in the most critical region: near the optimum. This phenomenon is known as the fidelity-performance correlation gap.

Globally, a 1-epoch training run correlates reasonably well with a 100-epoch training run because completely broken hyperparameter configurations fail immediately. However, Klein et al. empirically demonstrated that this correlation weakens significantly when focusing exclusively on the top-performing configurations.

This breakdown creates misleading low-fidelity optima. Because the low-fidelity landscape diverges from the high-fidelity landscape near the peak, an optimizer relying on the proxy will be steered toward entirely wrong regions of the parameter space.

Performance
  ^
  |          High-Fidelity Optimum
  |             _/\_
  |            /    \       Low-Fidelity Optimum
  |    _______/      \________/\_
  |   /                          \
  +-----------------------------------> Parameter Space

As documented by Wang et al. in the context of inertial confinement fusion design, traditional sequential approaches (exploring with low-fidelity, then refining with high-fidelity) fail because the optimal parameters identified by coarse simulations are often completely decoupled from the true physical optima.


Pathologies in Deep Learning Hyperparameter Tuning

In modern deep learning, multi-fidelity optimization is the bedrock of popular Hyperparameter Optimization (HPO) and Neural Architecture Search (NAS) frameworks like BOHB (Falkner et al.) and DEHB (Awad). These algorithms use early-stage pruning (e.g., Hyperband) to aggressively discard configurations that perform poorly on small resource budgets (short training epochs).

However, this strategy assumes that early performance monotonically predicts final performance. Micha Henheik demonstrated that this assumption frequently collapses when tuning complex machine learning models.

On intricate tasks, configurations that look highly promising in the first 5% of training can suffer from late-stage training instability, exploding gradients, or sudden model collapse during extended runs. Conversely, configurations that learn slowly at first might ultimately achieve superior generalization. By pruning based on short, low-fidelity trials, multi-fidelity optimizers systematically misidentify stable configurations, leading to suboptimal final models.


Designing for Robustness: Dynamic and Convergence-Aware Models

To prevent low-fidelity proxies from misleading the optimization process, recent research focuses on algorithms that actively monitor and adapt to cross-fidelity correlation.

1. Dynamic Proxy Evaluation

Instead of trusting a proxy blindly, frameworks can treat proxy reliability as an unknown variable to be learned on the fly. For instance, the ProxyBO framework introduced by Shen et al. dynamically measures the predictive power of zero-cost proxies during Neural Architecture Search. If the correlation between the proxy pool and the validated performance drops, the system automatically shifts its weight away from the proxies, preventing uninformative signals from damaging the search.

2. Convergence-Aware Modeling

Rather than treating different fidelities as static, isolated slices of an objective function, advanced frameworks model the relationship between fidelities as a continuous dynamical system.

Xing et al. developed CAMO (Convergence-Aware Multi-fidelity Optimization), which encodes the "tendency to converge" directly into a specialized Gaussian Process kernel. By learning the trajectory of how the low-fidelity approximation evolves into the high-fidelity target, the model can explicitly predict whether an early-stage configuration is likely to collapse or stabilize, bridging the fidelity-performance gap.

Summary

Multi-fidelity optimization is not a free lunch; it is a leveraged bet on cross-fidelity correlation. The speedup it provides is fundamentally bounded by how well the cheap proxy mirrors the relative trends of the expensive ground truth. When building or using these systems, engineers must look beyond global correlation metrics and explicitly safeguard their pipelines against the local correlation gaps that emerge near the optima.

Key Foundational Work

Minimax Approach to Variable Fidelity Data Interpolation (2017) — contributed 3 concept elements to this direction.
Inverse Modeling of Hydrologic Systems with Adaptive Multifidelity Markov Chain Monte Carlo Simulations (2018) — contributed 2 concept elements to this direction.
Multi-fidelity Bayesian Optimisation with Continuous Approximations (2017) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_surrogate_model_correlation_e0604573,
  title = {surrogate model correlation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/surrogate-model-correlation/rdr_e0604573098f45cdb2c5e78ed89b70cf},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_surrogate_model_correlation_e0604573_v1,
  title = {surrogate model correlation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/surrogate-model-correlation/rdr_e0604573098f45cdb2c5e78ed89b70cf/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake