Multi-Fidelity Optimization for Efficient Model Search

18 concepts13 papers1 proposal

The search for optimal machine learning configurations—whether hyperparameters, neural architectures, or prompt sequences—is often bottlenecked by the sheer cost of evaluation. Traditionally, finding the best model required training every candidate to completion on the full dataset. Multi-fidelity optimization breaks this bottleneck by replacing expensive, high-fidelity evaluations with cheaper, low-fidelity proxies.

The Intuition of Proxies

The core insight behind multi-fidelity optimization is that we do not need a perfect measurement to identify a poor candidate. As Bernd Bischl explains, low-cost proxies—such as training on a small subset of data or for a limited number of iterations—provide a signal strong enough to prune the search space. By discarding underperforming configurations early, computational resources can be concentrated on the most promising regions.

This approach relies on the assumption that performance at low fidelity correlates strongly with performance at high fidelity. Baek Seong-Eun et al. demonstrate this empirically in the context of LoRA hyperparameter search, finding that evaluations on a random 10% data sample maintain a correlation exceeding 0.85 with full-dataset performance. Similarly, Kevin Swersky et al. argue that small data subsets provide sufficient information to rule out poor settings, effectively "transferring" knowledge from cheap tasks to expensive ones.

Mechanisms for Resource Allocation

Multi-fidelity systems typically use a hierarchical or scheduled approach to manage the trade-off between cost and accuracy:

  1. Early Pruning: Algorithms like Hyperband use resource scheduling to terminate unpromising trials. Lennart Schneider et al. show that using validation sample size as a fidelity metric allows an optimizer to scale efficiently, only committing large budgets to the most competitive candidates.
  2. Successive Halving: Marcio Monteiro et al. utilize exponentially increasing subsets of data to generate early performance signals. This allows for the prediction of final performance without full-scale training.
  3. Surrogate Modeling: In Neural Architecture Search (NAS), Zhao Wei et al. combine multi-fidelity evaluations with surrogate models to navigate complex, multi-objective search spaces, drastically reducing the cost of finding high-performing architectures like U-Nets.

The efficiency gains from these mechanisms are substantial. Luca Franceschi et al. note that utilizing low-fidelity signals often provides a more significant reduction in total optimization time than increasing the complexity of the search algorithm itself.

Applications Across the Pipeline

While originally popularized for Hyperparameter Optimization (HPO), multi-fidelity strategies have expanded across the machine learning pipeline:

  • Model Merging: Tommaso Mencattini et al. apply these concepts to evolutionary model merging. By using Item Response Theory (IRT) on tiny data subsets, they achieve a 50x reduction in compute costs, enabling the evolution of Large Language Models (LLMs) on consumer-grade GPUs.
  • Deep Learning Frameworks: Systems like Auto-PyTorch, described by Lucas Zimmer et al., integrate multi-fidelity meta-learning to jointly optimize architectures and training hyperparameters for tabular data.
  • Complex Domains: In Multi-Label Classification, where training is inherently slow, Marcel Wever et al. demonstrate that multi-fidelity techniques are essential for mitigating evaluation bottlenecks.

The Path to Sustainable AI

Beyond raw speed, multi-fidelity optimization is a cornerstone of "Green AutoML." Alexander Tornede et al. argue that by truncating training cycles and using data subsets, researchers can significantly reduce the carbon footprint of automated search. To support this shift, Katharina Eggensperger et al. introduced HPOBench, a collection of reproducible benchmarks designed specifically to test how well optimization algorithms handle these varying levels of fidelity.

Ultimately, as Yi-Wei Chen et al. summarize, the transition from manual tuning to automated pipelines relies on these estimation techniques to make the search for the "perfect" model computationally feasible.

Key Foundational Work

Applying Model-Based Optimization to Hyperparameter Optimization in Machine Learning (2015) — contributed 2 concept elements to this direction.
Multi-Task Bayesian Optimization (2013) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_multi_fidelity_optimization_low_fidelity_3057ad25,
  title = {multi-fidelity optimization (low-fidelity proxy evaluation)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/multi-fidelity-optimization-low-fidelity-proxy-evaluation/rdr_3057ad25365948299c92fb3f5721e434},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_multi_fidelity_optimization_low_fidelity_3057ad25_v1,
  title = {multi-fidelity optimization (low-fidelity proxy evaluation)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/multi-fidelity-optimization-low-fidelity-proxy-evaluation/rdr_3057ad25365948299c92fb3f5721e434/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake