TL;DR
AMM (Adaptive Multi Mode) is a budget-aware optimizer router that uses cheap "online landscape probes" to identify function characteristics—like curvature, separability, or ruggedness—on the fly. By matching the problem's geometry to the right optimization engine (e.g., quasi-Newton for smooth areas, CMA-ES for rugged ones), AMM significantly outperforms state-of-the-art optimizers across a wide range of budgets and dimensions.
What Problem Does This Paper Address?
In the world of continuous black-box optimization, there is no "one size fits all" algorithm. A trust-region method might be king when you only have a budget of $10 \cdot D$ evaluations (where $D$ is the dimension), while a restarted CMA-ES (Evolution Strategy) usually dominates when you have $1000 \cdot D$.
The challenge is that the "best" algorithm depends on both the available budget and the hidden landscape of the function. Existing "wizard" optimizers (like Nevergrad’s NGOpt) typically choose a strategy based only on metadata like dimension and budget. They don't actually "look" at the function's shape before committing. AMM addresses this by introducing online probes: tiny, inexpensive tests performed during the run to determine which optimization engine's implicit assumptions actually hold true for the problem at hand.
Methods
AMM acts as a traffic controller for a pool of established optimization engines, including SLSQP (quasi-Newton), COBYQA (trust-region), STEP (coordinate search), and various flavors of CMA-ES.
The core innovation is the routing logic guided by probes:
- Online Probes: These are short sequences of function evaluations designed to detect specific features:
- Curvature: Is the local area smooth enough for a second-order method?
- Separability: Can the dimensions be optimized independently?
- Plateaus/Kinks: Is the optimizer stuck on a flat surface or hitting a non-differentiable ridge?
- Budget-Aware Routing: As seen in the flowchart below, AMM uses a decision tree that incorporates the probe results alongside the remaining budget to "route" the problem to the most promising engine.
- The "Finisher": If specialized routes don't solve the problem, AMM defaults to a robust "finisher" (a (1+1)-CMA-ES variant) to spend the remaining budget.
Datasets
The authors evaluated AMM using the BBOB (Black-Box Optimization Benchmarking) suite, the industry standard for noiseless optimization.
- Development Set: Instances 1–100 were used to calibrate thresholds and design the routing rules.
- Final Test Set: Held-out instances 101–110 (which the algorithm never "saw" during design) were used for the headline results.
- MA-BBOB: A more challenging benchmark involving affine combinations of functions.
- Robustness Tests: The authors also tested on transformed versions of BBOB (rotated, shifted, and box-constrained) to ensure the routing logic wasn't just "overfitting" the standard benchmarks.
Key Insights and Ideas
- Probes as Guards: Instead of guessing, AMM uses probes as "guards." For example, it only triggers a coordinate search if a probe confirms the function is likely separable. This prevents wasting budget on engines that are fundamentally ill-suited for the landscape.
- The Cost of Knowledge: Probes aren't free; they cost evaluations. AMM is designed so that probes only run if their cost is a small fraction of the total budget, ensuring that "looking before you leap" actually pays off.
- Routing > Engines: A key finding was that AMM's performance gain comes from the routing logic, not just having a large pool of engines. When the probes were disabled and the engines were run on a fixed schedule, performance dropped significantly.
- AI-Assisted Rigor: Interestingly, the author used LLMs (Claude) to propose "fixes" and write pre-registered predictions. This "development loop" ensured that every change to the algorithm had to prove its worth on unseen data before being adopted.
Key Results
AMM demonstrated a clear lead over a field of seven high-profile competitors, including IPOP-CMA-ES, SLSQP, and Nevergrad’s NGOpt.
- Target Share: In a massive comparison across 600 "cells" (combinations of function, dimension, and budget), AMM achieved the highest share of reached targets in 382 cells.
- Performance Gap: AMM outperformed IPOP-CMA-ES by a margin of +0.12 to +0.31 in target share across different budgets.
- Scaling: As shown in the figures below, AMM maintains its lead as the budget grows, effectively "switching gears" as more evaluations become available.
Limitations and Future Work
- Coordinate Sensitivity: Some of AMM's most effective routes (like coordinate search) are sensitive to the coordinate system. If a separable function is rotated, these routes fail, and AMM falls back to more general (but slower) engines.
- Anytime Performance: AMM is "budget-aware"—it plans its strategy based on a known end-point. While the authors tested an "anytime" mode that works without knowing the budget, it currently loses some efficiency compared to the budget-aware version.
- Noisy Landscapes: The current study focused on noiseless functions. Extending these online probes to handle stochastic noise is a logical next step.
- Computational Overhead: While the probes are "cheap" in terms of function evaluations, the internal logic of AMM adds some CPU overhead, which might be a factor if the objective function itself is extremely fast to evaluate.
Summary updated