Introduction
Neural Architecture Search (NAS) has emerged as a powerful tool for automating deep learning model design. To mitigate the prohibitive costs of training thousands of individual models, one-shot NAS utilizes a "supernet" with shared weights. However, this approach frequently encounters the co-adaptation problem: weights are optimized to satisfy a vast array of architectures simultaneously, preventing them from reaching an ideal state for any specific configuration. Consequently, the supernet becomes an unreliable proxy for ranking architectures.
While few-shot NAS attempts to address this by partitioning the supernet edge-by-edge, these methods often ignore inter-edge relationships and result in complex, computationally expensive search trees. This paper introduces HEP-NAS, a novel framework that employs a "hierarchy-wise" partitioning strategy combined with mutual distillation to enhance search efficiency and accuracy.
Methodology: Hierarchical Edge Partitioning and Mutual Distillation
The core of HEP-NAS lies in its structural approach to search space reduction, moving beyond isolated edge-wise splits to consider node-level context.
Hierarchical Edge Partitioning (HEP)
Unlike traditional methods that split one edge at a time, HEP-NAS groups all edges leading into a single destination node—termed a "hierarchy." By performing a Cartesian product of operation groups across these edges, the method creates sub-supernets that capture the optimal combination of operations for a node. This hierarchy-level partitioning acknowledges that an operation's performance on one edge is inherently linked to the operations on concurrent edges feeding the same node.
Search Space Mutual Distillation (SMD)
To stabilize the weight-sharing mechanism during partitioning, HEP-NAS introduces Search Space Mutual Distillation (SMD). Sub-supernets within a hierarchy engage in collaborative learning by sharing soft labels. Furthermore, they are guided by the top-performing sub-supernet from the previous hierarchy level. This dual-guidance system prevents performance drops during splitting and ensures the search does not diverge into local optima.
Experimental Results
HEP-NAS was evaluated on standard benchmarks including CIFAR-10, CIFAR-100, ImageNet, and NAS-Bench-201. The method consistently demonstrated superior performance and efficiency compared to state-of-the-art (SOTA) one-shot and few-shot NAS techniques.
Performance Benchmarks
- CIFAR-10 & CIFAR-100: HEP-NAS achieved a top-1 accuracy of 97.56% on CIFAR-10 and 83.17% on CIFAR-100, outperforming competitive methods like GM-NAS and few-shot DARTS.
- ImageNet Transferability: Architectures discovered on CIFAR-10 transferred effectively to ImageNet, reaching a 76.4% top-1 accuracy.
- NAS-Bench-201: On the ImageNet16-120 subset, HEP-NAS achieved 46.51% accuracy, nearing the theoretical optimum of 47.31% for that search space.
Efficiency
While maintaining high accuracy, HEP-NAS remains computationally viable. It achieves significant error rate reductions (e.g., 14% less than SNAS) while requiring substantially fewer GPU-days than many few-shot counterparts, such as GM ProxylessNAS.
Conclusion
HEP-NAS effectively addresses the co-adaptation challenge in NAS by leveraging hierarchical edge partitioning and mutual distillation. By focusing on node-level operation combinations, it captures critical inter-edge correlations that simpler methods overlook. While the search cost (0.8 to 1.5 GPU-days) is slightly higher than the fastest one-shot methods, the gains in accuracy and reliability make it a compelling solution for robust architecture discovery.
Summary updated