HOTNAS: Hierarchical Optimal Transport for Neural Architecture Search

Jiechao Yang, Yong Liu, Hongteng XuCVPR 2023 · DOI

Introduction

Neural Architecture Search (NAS) has emerged as a powerful tool for automating the design of deep learning models. However, exploring the vast space of potential architectures is computationally intensive. To mitigate this, researchers frequently utilize modular cell-based search spaces, where a network is constructed by stacking several "cells." A significant challenge in this paradigm is defining a robust similarity metric between architectures, especially when cells can have varying internal structures. Such metrics are vital for Bayesian Optimization (BO), which relies on surrogate models to predict the performance of untested designs based on their resemblance to known ones.

The HOTNAS Framework

The authors introduce HOTNAS (Hierarchical Optimal Transport for NAS), a novel approach that employs a two-tier similarity metric to compare architectures comprehensively.

The hierarchical similarity approach in HOTNAS, illustrating (a) cell-level similarity through pointwise and pairwise matching, and (b) network-level similarity considering cell types and path differences.
  1. Cell-level Similarity (iFGW): HOTNAS utilizes an "improved Fused Gromov-Wasserstein" (iFGW) metric. Unlike simpler metrics that only count operation types, iFGW accounts for the "information flow" and topological positions of nodes within a cell. It calculates the optimal cost of "transporting" one cell's structure into another.
  2. Network-level Similarity (HOTNN): Building upon cell similarities, the algorithm compares the global arrangement of cells within the network backbone. It explicitly considers the relative positions of cells, recognizing that a specific cell's impact may vary depending on its location in the hierarchy (e.g., early vs. late layers).
  3. Bayesian Optimization: These hierarchical distances are integrated into a Gaussian Process (GP) kernel, enabling the search algorithm to build a sophisticated surrogate model that understands the complex geometry of the architecture search space.

Experimental Evaluation

HOTNAS was evaluated on two prominent benchmarks:

  • TransNAS-Bench-101: A multi-task benchmark covering seven diverse computer vision tasks, including object classification, autoencoding, and semantic segmentation.
  • DARTS Search Space: Evaluated on the standard CIFAR-10 and CIFAR-100 image classification datasets.

Results and Discussion

HOTNAS consistently outperformed state-of-the-art NAS methods, including NASBOT, BANANAS, and evolutionary search. On TransNAS-Bench-101, it achieved lower validation and test errors across all seven tasks. In the DARTS space, it reached a test error of 2.43% on CIFAR-10 and 18.46% on CIFAR-100.

Convergence comparison showing the best validation loss versus the number of observations. HOTNAS demonstrates superior sample efficiency compared to random, evolutionary, and other BO-based search methods.

A key finding is the importance of "location difference." By penalizing differences in the global position of similar cells, HOTNAS makes more informed decisions, leading to significantly higher sample efficiency. As shown in the convergence plots, HOTNAS identifies high-performing models in fewer iterations than traditional search strategies.

Conclusion

HOTNAS demonstrates that a mathematically rigorous, hierarchical approach to architecture similarity can drastically improve the efficiency of NAS. By balancing internal cell topology with global network arrangement via Optimal Transport, it provides a powerful framework for discovering optimized models across a wide range of tasks. Future work may involve extending this approach to non-modular "macro" search spaces and further automating the tuning of its structural trade-off parameters.

Summary updated

BibTeX
@inproceedings{yang2023hotnas,
  author = {Jiechao Yang and Yong Liu and Hongteng Xu},
  title = {{HOTNAS: Hierarchical Optimal Transport for Neural Architecture Search}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52729.2023.01154},
}
Figure