DyESP: Accelerating Hyperparameter-Architecture Search via Dynamic Exploration and Space Pruning

Xukun Liu, Haoze Lv, Fenglong Ma, Chi Wang, Dongkuan Xu

Abstract

Hyperparameter-Architecture Search (HAS) is a critical but computationally expensive task in deep learning. This paper introduces DyESP, a framework that accelerates HAS by employing a meta-learning scheduler for dynamic exploration and a surrogate-based pruner for early identification of sub-optimal configurations. DyESP achieves up to 71x speedup over standard baselines while maintaining or improving model accuracy across diverse benchmarks.

Introduction

Optimizing deep learning models typically involves two distinct phases: Neural Architecture Search (NAS) and Hyperparameter Optimization (HPO). While joint search (HAS) offers superior performance, the resulting search space is significantly larger, often leading to prohibitive computational costs. Standard search algorithms frequently utilize static exploration strategies that fail to adapt to the specific characteristics of the search space or the progress of the search itself.

The DyESP Framework

The DyESP framework addresses these inefficiencies through two primary components: a Meta-Scheduler and a Hyperparameter Pruner.

The DyESP framework architecture, highlighting the interaction between the Meta-Scheduler, Search Algorithm, and Hyperparameter Pruner.

Meta-Scheduler and Dynamic Exploration

The Meta-Scheduler is implemented as a three-layer Multi-Layer Perceptron (MLP) that dynamically adjusts the Exploration Degree ($M$) based on the "Reward History" of previous trials. By utilizing meta-learning, the scheduler extracts common knowledge across different search spaces, allowing it to decide whether to broaden the search or focus on promising regions. This adaptive approach prevents the search from getting stuck in local optima and avoids wasted computation on unpromising areas.

Dynamic adjustment of the exploration degree over the course of trials on the NB201 benchmark.

Hyperparameter Pruner

To further reduce costs, DyESP incorporates a surrogate model that acts as a pruner. This model utilizes low-cost static metrics, such as FLOPs and model size, to predict the performance gap between configurations without requiring full training. Configurations predicted to be inferior are discarded early, significantly saving GPU resources.

Experimental Evaluation

DyESP was evaluated on a wide range of benchmarks, including NASBench101, NASBench201, and JHAS-Bench-201 (covering CIFAR-10, Fashion MNIST, and Colorectal Histology).

Key Results

  • Acceleration: DyESP achieved a 71.4x speedup on NASBench201 and a 40.3x speedup on the NATS benchmark compared to Random Search.
  • Efficiency: On NASBench101, DyESP reached the top performance tier in 432 trials, whereas Genetic Algorithms failed to reach the same level after 5,000 trials.
  • Versatility: The framework demonstrated consistent performance gains across diverse tasks, including image classification, speech recognition, and graph neural networks.

Key Insights

The success of DyESP highlights several important principles in AutoML:

  1. Adaptive Strategies: There is no "one-size-fits-all" search strategy; different spaces require different exploration curves.
  2. Proxy Metrics: Static architectural properties are powerful indicators of performance and can be effectively used for early-stage pruning.
  3. Feedback Loops: Integrating pruning results back into the meta-scheduler allows for continuous refinement of the sampling strategy.

Conclusion

DyESP represents a significant advancement in HAS by making the search process both faster and more intelligent. By combining dynamic meta-learning with efficient space pruning, it overcomes the computational barriers of joint architecture and hyperparameter optimization. Future work will explore more complex architectures for the scheduler and the integration of full computation graphs into the pruning process.

Summary updated

Abstract

In this work, we introduce DyESP, a novel approach that unites dynamic exploration with space pruning to expedite the combined search of hyperparameters and architecture, enhancing the efficiency and accuracy of hyperparameter-architecture search (HAS). Central to DyESP are two innovative components: a meta-scheduler that customizes the search strategy for varying spaces and a pruner designed to minimize the hyperparameter space by discarding suboptimal configurations. The meta-scheduler leverages historical data to dynamically refine the search direction, targeting the most promising areas while minimizing unnecessary exploration. Meanwhile, the pruner employs a surrogate model, specifically a fine-tuned multilayer perceptron (MLP), to predict and eliminate inferior configurations based on static metrics, thereby streamlining the search and conserving computational resources. The results from the pruner, which identifies and removes underperforming configurations, are fed into the meta-scheduler. This process updates the historical dataset used by the meta-scheduler, enabling it to adjust the exploration degree and refine the sampling strategy for subsequent iterations. This integration ensures the meta-scheduler is continually updated with relevant data, allowing for more accurate and timely adjustments to the exploration strategy. Experiments on various benchmarks show that DyESP outperforms existing methods in terms of both speed and stability on almost all benchmarks.
BibTeX
@inproceedings{liu2025dyesp,
  author = {Xukun Liu and Haoze Lv and Fenglong Ma and Chi Wang and Dongkuan Xu},
  title = {{DyESP: Accelerating Hyperparameter-Architecture Search via Dynamic Exploration and Space Pruning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaaiss.v5i1.35585},
}
Figure