Single-Path NAS: Designing Hardware-Efficient ConvNets in Less Than 4 Hours

Dimitrios Stamoulis, Ruizhou Ding, Di Wang, Dimitrios Lymberopoulos, Bodhi Priyantha, Jie Liu, Diana Marculescu

Abstract

Can we automatically design a Convolutional Network (ConvNet) with the highest image classification accuracy under the latency constraint of a mobile device? Neural architecture search (NAS) has revolutionized the design of hardware-efficient ConvNets by automating this process. However, the NAS problem remains challenging due to the combinatorially large design space, causing a significant searching time (at least 200 GPU-hours). To alleviate this complexity, we propose Single-Path NAS, a novel differentiable NAS method for designing hardware-efficient ConvNets in less than 4 hours. Our contributions include a single-path search space that drastically decreases trainable parameters, achieving state-of-the-art 74.96% top-1 accuracy on ImageNet with 79 ms latency on a Pixel 1 phone. The search cost is only 8 epochs (30 TPU-hours), up to 5,000x faster than prior work.

Introduction

Designing Convolutional Neural Networks (ConvNets) for mobile devices is a delicate balancing act between maximizing accuracy and adhering to strict hardware constraints like latency and power. Neural Architecture Search (NAS) was developed to automate this design process, but it traditionally suffers from a "combinatorial explosion." For instance, a network with 22 layers and 5 choices per layer yields approximately $10^{15}$ possible architectures. Early NAS methods relied on Reinforcement Learning to sample thousands of these, which was computationally prohibitive. While recent differentiable NAS methods improved efficiency by building a "supernet" containing all possible paths, they remain memory-intensive and require complex bi-level optimization.

Methodology: The Single-Path Search Space

The core innovation of this work is the Single-path search space. Unlike previous differentiable NAS methods that create a "multi-path" supernet where each candidate operation (e.g., a 3x3 kernel vs. a 5x5 kernel) is a separate branch, Single-Path NAS uses a single-path over-parameterized ConvNet.

Comparison between multi-path and single-path search spaces.

As shown in Figure 1, the proposed method encodes all architectural decisions with shared convolutional kernel parameters. The researchers observed that a 3x3 convolution is essentially the "inner core" of a 5x5 convolution. By using a single 5x5 kernel and an indicator function to decide whether to use the "outer shell," they can encode the choice of kernel size directly into the weights of a single path. This "superkernel" approach drastically reduces the number of trainable parameters and the search cost.

Architecture and Optimization

The search space is built upon a fixed macro-architecture inspired by MobileNetV2, consisting of multiple blocks of Mobile Inverted Bottleneck (MBConv) layers. Within each block, the method searches for the optimal kernel size and expansion ratio using the searchable superkernel.

Overview of the Single-Path NAS macro-architecture and superkernel design.

Key technical components include:

  • Trainable Indicator Functions: Group Lasso terms are used to determine the importance of different weight subsets, such as the outer shell of a kernel or additional channels for expansion ratios.
  • Sigmoid Relaxation: To make discrete architectural choices searchable via gradient descent, the indicator functions are relaxed into continuous sigmoid functions during the search phase.
  • Hardware-Aware Loss: The objective function combines standard cross-entropy loss with a differentiable latency term, based on a runtime model profiled on a Google Pixel 1.

Key Results

Single-Path NAS demonstrates exceptional efficiency and performance:

  • Efficiency: The search process takes only 8 epochs, totaling approximately 3.75 hours on a TPUv2. This is 5,000x faster than MnasNet and 25x faster than ProxylessNAS.
  • Accuracy: On the ImageNet dataset, the resulting model achieved 74.96% top-1 accuracy.
  • Hardware Performance: The model runs in 79.48ms on a Pixel 1, outperforming other NAS-designed models (like FBNet and MnasNet) in the same latency bracket by 0.3% to 0.8% accuracy.

Conclusion

Single-Path NAS represents a significant leap in NAS efficiency by reimagining the search space as a single path of shared weights. While the current implementation focuses on searching for kernel sizes and expansion ratios within a fixed backbone, it provides a powerful framework for rapid hardware-efficient design. To support further research, the authors have open-sourced their entire codebase.

Summary updated

Abstract

Can we automatically design a Convolutional Network (ConvNet) with the highest image classification accuracy under the latency constraint of a mobile device? Neural architecture search (NAS) has revolutionized the design of hardware-efficient ConvNets by automating this process. However, the NAS problem remains challenging due to the combinatorially large design space, causing a significant searching time (at least 200 GPU-hours). To alleviate this complexity, we propose Single-Path NAS, a novel differentiable NAS method for designing hardware-efficient ConvNets in less than 4 h. Our contributions are as follows: 1. Single-path search space: Compared to previous differentiable NAS methods, Single-Path NAS uses one single-path over-parameterized ConvNet to encode all architectural decisions with shared convolutional kernel parameters, hence drastically decreasing the number of trainable parameters and the search cost down to few epochs. 2. Hardware-efficient ImageNet classification: Single-Path NAS achieves $$74.96\%$$ top-1 accuracy on ImageNet with 79 ms latency on a Pixel 1 phone, which is state-of-the-art accuracy compared to NAS methods with similar inference latency constraints ($$\le $$80 ms). 3. NAS efficiency: Single-Path NAS search cost is only 8 epochs (30 TPU-hours), which is up to 5,000$$\times $$ faster compared to prior work. 4. Reproducibility: Unlike all recent mobile-efficient NAS methods which only release pretrained models, we open-source our entire codebase at: https://github.com/dstamoulis/single-path-nas .
BibTeX
@inproceedings{stamoulis2019single,
  author = {Dimitrios Stamoulis and Ruizhou Ding and Di Wang and Dimitrios Lymberopoulos and Bodhi Priyantha and Jie Liu and Diana Marculescu},
  title = {{Single-Path NAS: Designing Hardware-Efficient ConvNets in Less Than 4 Hours}},
  booktitle = {ECML/PKDD},
  year = {2019},
  url = {https://openreview.net/forum?id=kdar8J7Hzgx},
}
Figure