Introduction
Neural Architecture Search (NAS) has emerged as a powerful tool for automating the design of deep learning models. However, exploring the vast space of potential architectures is computationally intensive. To mitigate this, researchers frequently utilize modular cell-based search spaces, where a network is constructed by stacking several "cells." A significant challenge in this paradigm is defining a robust similarity metric between architectures, especially when cells can have varying internal structures. Such metrics are vital for Bayesian Optimization (BO), which relies on surrogate models to predict the performance of untested designs based on their resemblance to known ones.
The HOTNAS Framework
The authors introduce HOTNAS (Hierarchical Optimal Transport for NAS), a novel approach that employs a two-tier similarity metric to compare architectures comprehensively.
- Cell-level Similarity (iFGW): HOTNAS utilizes an "improved Fused Gromov-Wasserstein" (iFGW) metric. Unlike simpler metrics that only count operation types, iFGW accounts for the "information flow" and topological positions of nodes within a cell. It calculates the optimal cost of "transporting" one cell's structure into another.
- Network-level Similarity (HOTNN): Building upon cell similarities, the algorithm compares the global arrangement of cells within the network backbone. It explicitly considers the relative positions of cells, recognizing that a specific cell's impact may vary depending on its location in the hierarchy (e.g., early vs. late layers).
- Bayesian Optimization: These hierarchical distances are integrated into a Gaussian Process (GP) kernel, enabling the search algorithm to build a sophisticated surrogate model that understands the complex geometry of the architecture search space.
Experimental Evaluation
HOTNAS was evaluated on two prominent benchmarks:
- TransNAS-Bench-101: A multi-task benchmark covering seven diverse computer vision tasks, including object classification, autoencoding, and semantic segmentation.
- DARTS Search Space: Evaluated on the standard CIFAR-10 and CIFAR-100 image classification datasets.
Results and Discussion
HOTNAS consistently outperformed state-of-the-art NAS methods, including NASBOT, BANANAS, and evolutionary search. On TransNAS-Bench-101, it achieved lower validation and test errors across all seven tasks. In the DARTS space, it reached a test error of 2.43% on CIFAR-10 and 18.46% on CIFAR-100.
A key finding is the importance of "location difference." By penalizing differences in the global position of similar cells, HOTNAS makes more informed decisions, leading to significantly higher sample efficiency. As shown in the convergence plots, HOTNAS identifies high-performing models in fewer iterations than traditional search strategies.
Conclusion
HOTNAS demonstrates that a mathematically rigorous, hierarchical approach to architecture similarity can drastically improve the efficiency of NAS. By balancing internal cell topology with global network arrangement via Optimal Transport, it provides a powerful framework for discovering optimized models across a wide range of tasks. Future work may involve extending this approach to non-modular "macro" search spaces and further automating the tuning of its structural trade-off parameters.
Summary updated