Hyperparameter optimization (HPO) is frequently the most computationally expensive phase of the machine learning pipeline. Standard black-box optimization algorithms often treat the search space uniformly, evaluating configurations without considering their computational footprint. This blind spot can lead to massive resource waste early in the search, as optimizers spend valuable hours evaluating heavy, poorly performing configurations.
To combat this, a powerful paradigm has emerged: cost-prioritized initialization and progressive scaling. The core intuition is simple: start the search in low-cost, low-fidelity, or low-dimensional regions of the parameter space, and only transition to expensive configurations when there is clear evidence of performance gains.
The Core Intuition: Leverage the Cheap to Inform the Expensive
Why can we trust cheap evaluations to guide expensive ones? The strategy works because optimal hyperparameter configurations often exhibit scale invariance. For instance, a learning rate, regularization strength, or architectural layout that performs well on a low-resolution image or a downscaled version of a model often provides an excellent starting point for the full-scale problem.
By exploiting this correlation, we can map out the rough topography of the optimization landscape at a fraction of the cost, using those insights to prune away bad regions before committing heavy computational resources.
Restricting Search to Admissible Low-Cost Regions
One way to implement this intuition is to explicitly bias the optimizer toward inexpensive configurations during the initial phases of the search.
In Frugal Optimization for Cost-related Hyperparameters, Wang et al. introduce Cost-Frugal Optimization (CFO). Instead of building complex, overhead-heavy cost-prediction models, the authors initialize a randomized direct-search algorithm at a known low-complexity starting point. The search moves cautiously outward into more expensive regions of the parameter space only if those configurations demonstrate clear performance improvements over the cheaper baselines.
To scale this philosophy to global search algorithms, Wang et al. developed BlendSearch in ECONOMIC HYPERPARAMETER OPTIMIZATION WITH BLENDED SEARCH STRATEGY. BlendSearch uses a configuration validator to restrict early search suggestions to a subset of the parameter space called an admissible region. This region is initially heavily biased toward low-cost evaluations. As the optimization progresses and the algorithm gathers data, the admissible region dynamically expands. This prevents the premature exhaustion of resources on expensive, low-quality configurations, making high-performance tuning accessible even under tight hardware constraints.
[ Full Search Space ]
+-----------------------------------------+
| Expensive Configurations |
| +---------------------------------+ |
| | Admissible Region (Expanded) | |
| | +-------------------------+ | |
| | | Admissible (Initial) | | |
| | | [Low-Cost Start] | | |
| | +-------------------------+ | |
| +---------------------------------+ |
+-----------------------------------------+
Search expands outward only as
performance gains are validated.
Exploiting Resolution and Fidelity Invariance
Instead of just restricting the parameter space, we can also alter the data fidelity itself to save costs. This is particularly effective in computer vision, where image resolution directly dictates training time.
In Speeding up the Hyperparameter Optimization of Deep Convolutional Neural Networks, Hinz et al. leverage the fact that the relative importance and optimal values of hyperparameters remain highly consistent across different image resolutions for a given dataset.
The authors introduce a method that optimizes hyperparameters progressively:
- Low-Resolution Phase: Run HPO on downscaled images to rapidly discover a promising search space at minimal computational cost.
- Progression Phase: Gradually increase the image resolution.
- High-Resolution Phase: Use the boundaries and insights discovered in the low-resolution phase to constrain and guide the search on full-fidelity images.
This multi-stage approach allows for fast convergence to effective hyperparameter settings, drastically reducing total wall-clock time without compromising the final model's performance.
Budget-Aware Trust Regions
When managing high-dimensional or mixed search spaces, transitioning from low-cost approximations to high-fidelity spaces introduces a risk: the optimizer can become trapped. If an algorithm uses a fixed strategy to shrink its search area (or "trust region"), it might over-refine a low-dimensional approximation and run out of budget before it ever explores the true, high-fidelity space.
To solve this problem, Papenmeier et al. developed Bounce in Bounce: Reliable High-Dimensional Bayesian Optimization for Combinatorial and Mixed Spaces. The authors dynamically adjust trust-region sizes based on the remaining evaluation budget allocated to a specific subspace.
By explicitly linking trust-region shrinkage to the remaining budget, the algorithm prevents the search from prematurely locking into early, low-dimensional approximations. This budget-aware scaling ensures that the computational investment is distributed effectively across different levels of dimensionality, allowing the optimizer to reliably transition to high-fidelity representations when it matters most.
Key Foundational Work
Speeding up the Hyperparameter Optimization of Deep Convolutional Neural Networks (2018) — contributed 8 concept elements to this direction.
Frugal Optimization for Cost-related Hyperparameters (2021) — contributed 3 concept elements to this direction.
ECONOMIC HYPERPARAMETER OPTIMIZATION WITH BLENDED SEARCH STRATEGY (2021) — contributed 2 concept elements to this direction.