Abstract
Hyperparameter-Architecture Search (HAS) is a critical but computationally expensive task in deep learning. This paper introduces DyESP, a framework that accelerates HAS by employing a meta-learning scheduler for dynamic exploration and a surrogate-based pruner for early identification of sub-optimal configurations. DyESP achieves up to 71x speedup over standard baselines while maintaining or improving model accuracy across diverse benchmarks.
Introduction
Optimizing deep learning models typically involves two distinct phases: Neural Architecture Search (NAS) and Hyperparameter Optimization (HPO). While joint search (HAS) offers superior performance, the resulting search space is significantly larger, often leading to prohibitive computational costs. Standard search algorithms frequently utilize static exploration strategies that fail to adapt to the specific characteristics of the search space or the progress of the search itself.
The DyESP Framework
The DyESP framework addresses these inefficiencies through two primary components: a Meta-Scheduler and a Hyperparameter Pruner.
Meta-Scheduler and Dynamic Exploration
The Meta-Scheduler is implemented as a three-layer Multi-Layer Perceptron (MLP) that dynamically adjusts the Exploration Degree ($M$) based on the "Reward History" of previous trials. By utilizing meta-learning, the scheduler extracts common knowledge across different search spaces, allowing it to decide whether to broaden the search or focus on promising regions. This adaptive approach prevents the search from getting stuck in local optima and avoids wasted computation on unpromising areas.
Hyperparameter Pruner
To further reduce costs, DyESP incorporates a surrogate model that acts as a pruner. This model utilizes low-cost static metrics, such as FLOPs and model size, to predict the performance gap between configurations without requiring full training. Configurations predicted to be inferior are discarded early, significantly saving GPU resources.
Experimental Evaluation
DyESP was evaluated on a wide range of benchmarks, including NASBench101, NASBench201, and JHAS-Bench-201 (covering CIFAR-10, Fashion MNIST, and Colorectal Histology).
Key Results
- Acceleration: DyESP achieved a 71.4x speedup on NASBench201 and a 40.3x speedup on the NATS benchmark compared to Random Search.
- Efficiency: On NASBench101, DyESP reached the top performance tier in 432 trials, whereas Genetic Algorithms failed to reach the same level after 5,000 trials.
- Versatility: The framework demonstrated consistent performance gains across diverse tasks, including image classification, speech recognition, and graph neural networks.
Key Insights
The success of DyESP highlights several important principles in AutoML:
- Adaptive Strategies: There is no "one-size-fits-all" search strategy; different spaces require different exploration curves.
- Proxy Metrics: Static architectural properties are powerful indicators of performance and can be effectively used for early-stage pruning.
- Feedback Loops: Integrating pruning results back into the meta-scheduler allows for continuous refinement of the sampling strategy.
Conclusion
DyESP represents a significant advancement in HAS by making the search process both faster and more intelligent. By combining dynamic meta-learning with efficient space pruning, it overcomes the computational barriers of joint architecture and hyperparameter optimization. Future work will explore more complex architectures for the scheduler and the integration of full computation graphs into the pruning process.
Summary updated