Abstract
In addition to the best model architecture and hyperparameters, a full AutoML solution requires selecting appropriate hardware automatically. This can be framed as a multi-objective optimization problem: there is not a single best hardware configuration but a set of optimal ones achieving different trade-offs between cost and runtime. In practice, some choices may be overly costly or take days to train. To lift this burden, we adopt a multi-objective approach that selects and adapts the hardware configuration automatically alongside neural architectures and their hyperparameters. Our method builds on Hyperband and extends it in two ways. First, we replace the stopping rule used in Hyperband by a non-dominated sorting rule to preemptively stop unpromising configurations. Second, we leverage hyperparameter evaluations from related tasks via transfer learning by building a probabilistic estimate of the Pareto front that finds promising configurations more efficiently than random search. We show in extensive NAS and HPO experiments that both ingredients bring significant speed-ups and cost savings, with little to no impact on accuracy. In three benchmarks where hardware is selected in addition to hyperparameters, we obtain runtime and cost reductions of at least 5.8x and 8.8x, respectively. Furthermore, when applying our multi-objective method to the tuning of hyperparameters only, we obtain a 10% improvement in runtime while maintaining the same accuracy on two popular NAS benchmarks.
Introduction
Most Neural Architecture Search (NAS) and Hyperparameter Optimization (HPO) methods assume the hardware is a fixed constant. However, selecting the appropriate hardware is a critical part of a full AutoML solution. This paper argues that we should treat hardware selection as a tunable parameter alongside the model itself, framing the challenge as a multi-objective optimization problem. By extending the Hyperband algorithm with non-dominated sorting and transfer learning, the authors achieve significant cost reductions and speedups with negligible impact on model accuracy.
The Problem: Hardware-First Limitations
Traditional "hardware-first" approaches, where a specific instance (e.g., an AWS GPU instance) is chosen before the search begins, suffer from two major flaws:
- The Pareto Problem: There is rarely a single "best" hardware. One instance might be faster but significantly more expensive, while another is slow but cheap.
- Non-Stationarity: Hardware performance depends on the model's hyperparameters. For example, a specific GPU might be highly efficient for small batch sizes but fall off a "performance cliff" or run out of memory as the batch size increases.
Methodology
The authors build upon Hyperband, an early-stopping algorithm, and introduce two key innovations to handle multiple objectives such as accuracy, cost, and runtime:
- Non-Dominated Sorting (+ND): Instead of using "scalarization" (merging multiple metrics into one), this method uses non-dominated sorting to identify the Pareto front—the set of configurations where one objective cannot be improved without degrading another. A "greedy epsilon-net" strategy is employed to ensure the search covers a diverse range of trade-offs.
- Transfer Learning for Sampling (+tr): To avoid random sampling, the authors leverage data from previous, related tasks. A probabilistic surrogate model estimates the likely location of the Pareto front, allowing the search to focus on high-potential areas immediately.
Experimental Setup
The approach was evaluated on five benchmarks:
- Standard NAS Benchmarks:
nas201andfcnet. - Hardware-Aware Benchmark:
hw-nas, covering 15,625 architectures across six edge devices. - Cloud Benchmarks: The authors introduced
fcnet-cloudandnas201-cloud, which include measured runtimes and costs for 37 different AWS instance types.
Key Insights and Results
A significant insight is that hardware performance is highly sensitive to hyperparameters like batch size. A configuration that is optimal at one batch size may be completely dominated at another.
The multi-objective approach yielded substantial efficiency gains:
- Joint Hardware + HPO Tuning: On cloud benchmarks, the proposed method (+ND+tr) achieved runtime reductions of at least 5.8x and cost reductions of at least 8.8x compared to standard Hyperband, with similar error rates.
- HPO-only Tuning: Even without hardware tuning, the multi-objective sorting provided a 10% runtime improvement by more effectively pruning slow, low-accuracy configurations.
- Transfer Learning: This component was the primary driver of cost savings, enabling the algorithm to avoid expensive, slow hardware configurations early in the search.
Conclusion
Treating hardware as a first-class hyperparameter allows for significant cost and time savings in AutoML. By leveraging non-dominated sorting and transfer learning, researchers can navigate the complex trade-offs between accuracy, runtime, and cost more effectively than traditional single-objective or scalarized methods.
Summary updated