Adaptive Multi Mode: Budget-Aware Routing of Optimizers By Online Landscape Probes

Benjamin Jacob BodnerarXiv 2026 · arXiv

TL;DR

AMM (Adaptive Multi Mode) is a budget-aware optimizer router that uses cheap "online landscape probes" to identify function characteristics—like curvature, separability, or ruggedness—on the fly. By matching the problem's geometry to the right optimization engine (e.g., quasi-Newton for smooth areas, CMA-ES for rugged ones), AMM significantly outperforms state-of-the-art optimizers across a wide range of budgets and dimensions.

What Problem Does This Paper Address?

In the world of continuous black-box optimization, there is no "one size fits all" algorithm. A trust-region method might be king when you only have a budget of $10 \cdot D$ evaluations (where $D$ is the dimension), while a restarted CMA-ES (Evolution Strategy) usually dominates when you have $1000 \cdot D$.

The challenge is that the "best" algorithm depends on both the available budget and the hidden landscape of the function. Existing "wizard" optimizers (like Nevergrad’s NGOpt) typically choose a strategy based only on metadata like dimension and budget. They don't actually "look" at the function's shape before committing. AMM addresses this by introducing online probes: tiny, inexpensive tests performed during the run to determine which optimization engine's implicit assumptions actually hold true for the problem at hand.

Methods

AMM acts as a traffic controller for a pool of established optimization engines, including SLSQP (quasi-Newton), COBYQA (trust-region), STEP (coordinate search), and various flavors of CMA-ES.

The core innovation is the routing logic guided by probes:

  • Online Probes: These are short sequences of function evaluations designed to detect specific features:
  • Curvature: Is the local area smooth enough for a second-order method?
  • Separability: Can the dimensions be optimized independently?
  • Plateaus/Kinks: Is the optimizer stuck on a flat surface or hitting a non-differentiable ridge?
  • Budget-Aware Routing: As seen in the flowchart below, AMM uses a decision tree that incorporates the probe results alongside the remaining budget to "route" the problem to the most promising engine.
  • The "Finisher": If specialized routes don't solve the problem, AMM defaults to a robust "finisher" (a (1+1)-CMA-ES variant) to spend the remaining budget.
Figure 2: Routing of AMM (simplified). Rounded boxes are phases, diamonds are probes or gates, square boxes are engines; a dashed box groups the engines that spend the rest of the budget. Guards use the symbols of Table 2; “lq2 populations” are populations ranked by a quadratic surrogate. Not shown: the prefix’s single SLSQP descents (at 𝐷≤2 up to 30·𝐷; below min(𝑘des𝐷, 200)), its COBYQA hand-overs and the conditions of its lq2 populations (Algorithm 2); and, on a rugged reading with 𝑏< 𝑐𝑟𝐷, the capped BFGS try of 100·𝐷at 𝐷≤9, 𝐵≤500𝐷and 𝑏≥200𝐷, which rule (vii) replaces at 0.5 ≤𝛽< 1 by an immediate start of the finisher’s populations and rule (vi) continues to the end of the budget at a kink (Algorithm 1, lines 27–29).

Datasets

The authors evaluated AMM using the BBOB (Black-Box Optimization Benchmarking) suite, the industry standard for noiseless optimization.

  • Development Set: Instances 1–100 were used to calibrate thresholds and design the routing rules.
  • Final Test Set: Held-out instances 101–110 (which the algorithm never "saw" during design) were used for the headline results.
  • MA-BBOB: A more challenging benchmark involving affine combinations of functions.
  • Robustness Tests: The authors also tested on transformed versions of BBOB (rotated, shifted, and box-constrained) to ensure the routing logic wasn't just "overfitting" the standard benchmarks.

Key Insights and Ideas

  • Probes as Guards: Instead of guessing, AMM uses probes as "guards." For example, it only triggers a coordinate search if a probe confirms the function is likely separable. This prevents wasting budget on engines that are fundamentally ill-suited for the landscape.
  • The Cost of Knowledge: Probes aren't free; they cost evaluations. AMM is designed so that probes only run if their cost is a small fraction of the total budget, ensuring that "looking before you leap" actually pays off.
  • Routing > Engines: A key finding was that AMM's performance gain comes from the routing logic, not just having a large pool of engines. When the probes were disabled and the engines were run on a fixed schedule, performance dropped significantly.
  • AI-Assisted Rigor: Interestingly, the author used LLMs (Claude) to propose "fixes" and write pre-registered predictions. This "development loop" ensured that every change to the algorithm had to prove its worth on unseen data before being adopted.

Key Results

AMM demonstrated a clear lead over a field of seven high-profile competitors, including IPOP-CMA-ES, SLSQP, and Nevergrad’s NGOpt.

  • Target Share: In a massive comparison across 600 "cells" (combinations of function, dimension, and budget), AMM achieved the highest share of reached targets in 382 cells.
  • Performance Gap: AMM outperformed IPOP-CMA-ES by a margin of +0.12 to +0.31 in target share across different budgets.
  • Scaling: As shown in the figures below, AMM maintains its lead as the budget grows, effectively "switching gears" as more evaluations become available.
Figure 1: Held-out bbob instances 101–110 (f1–f24), equal budgets; field runs, SLSQP included, restart until the budget is spent or the final target is reached. Points: share of 51 targets reached per run, averaged over 10 instances and 120 function–dimension cells (𝐷= 2–40). HCMA (published COCO archive, own instances): 𝐷= 2–20 only; there AMM is level at 10·𝐷(+.016) and leads by +.10 to +.17 from 30·𝐷. TPE ran every 𝐷only at 10·𝐷. NADE (2025 MA-BBOB winner) and ERADS (van Stein’s 2024 algorithm): authors’ code, run after the final test. Against the pre-registered field AMM is highest in 382 of 600 function–dimension–budget cells (236 outright, 146 tied; 380, 234 outright, with NADE and ERADS added) and leads IPOP-CMA-ES by +.121 to +.312.
Figure 3 breaks down performance by dimension (D=2, 10, 40), showing AMM's robust lead across different problem scales.

Limitations and Future Work

  • Coordinate Sensitivity: Some of AMM's most effective routes (like coordinate search) are sensitive to the coordinate system. If a separable function is rotated, these routes fail, and AMM falls back to more general (but slower) engines.
  • Anytime Performance: AMM is "budget-aware"—it plans its strategy based on a known end-point. While the authors tested an "anytime" mode that works without knowing the budget, it currently loses some efficiency compared to the budget-aware version.
  • Noisy Landscapes: The current study focused on noiseless functions. Extending these online probes to handle stochastic noise is a logical next step.
  • Computational Overhead: While the probes are "cheap" in terms of function evaluations, the internal logic of AMM adds some CPU overhead, which might be a factor if the objective function itself is extremely fast to evaluate.

Summary updated

Abstract

The best classical optimizer for a continuous black-box problem depends on its budget and landscape. Existing routers decide from the dimension and budget, past improvement or features fitted to one benchmark, and few test an optimizer's implicit assumptions online. AMM (Adaptive Multi Mode) routes a known budget among quasi-Newton, trust-region, coordinate, multistart and evolution-strategy optimization engines with online probes of curvature, plateaus, separability, kinks, basins and fractal structure. Each probe has a default, a dimension-by-budget validity range and a cost charged to the budget. In its pre-registered development loop on bbob, 32 of 36 predictions held over the last 16 confirmations. AMM was then tested once on held-out bbob instances 101-110 (D=2-40, 10*D-1000*D) against up to seven competitors. It has the highest target share in 382 of 600 function-dimension-budget cells (236 outright, 146 tied) and, by a descriptive posterior rule, is credibly behind in 35, mostly at 10*D and 30*D. At every budget it also leads the controls built from its own engines (the best engine alone, the best engine per dimension and budget, and a greedy schedule), narrowly at 10*D and 30*D, and nevergrad's NgIohTuned. In pre-registered runs its lead over IPOP-CMA-ES holds under affine and box transformations, on box-constrained bbob and, pooled, on rotated f2-f4, but rotating the separable f3 and f4 removes most of the lead there. On held-out MA-BBOB its final share at 2000*D is level with IPOP-CMA-ES's and its anytime area lower. We published AMM as an open-source Python package (pip install amm-optimizer; https://github.com/BjBodner/amm-optimizer).
BibTeX
@misc{bodner2026adaptive,
  author = {Benjamin Jacob Bodner},
  title = {{Adaptive Multi Mode: Budget-Aware Routing of Optimizers By Online Landscape Probes}},
  year = {2026},
  eprint = {2611.06879},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.06879},
}
Figure