Abstract
Can we automatically design a Convolutional Network (ConvNet) with the highest image classification accuracy under the latency constraint of a mobile device? Neural architecture search (NAS) has revolutionized the design of hardware-efficient ConvNets by automating this process. However, the NAS problem remains challenging due to the combinatorially large design space, causing a significant searching time (at least 200 GPU-hours). To alleviate this complexity, we propose Single-Path NAS, a novel differentiable NAS method for designing hardware-efficient ConvNets in less than 4 hours. Our contributions include a single-path search space that drastically decreases trainable parameters, achieving state-of-the-art 74.96% top-1 accuracy on ImageNet with 79 ms latency on a Pixel 1 phone. The search cost is only 8 epochs (30 TPU-hours), up to 5,000x faster than prior work.
Introduction
Designing Convolutional Neural Networks (ConvNets) for mobile devices is a delicate balancing act between maximizing accuracy and adhering to strict hardware constraints like latency and power. Neural Architecture Search (NAS) was developed to automate this design process, but it traditionally suffers from a "combinatorial explosion." For instance, a network with 22 layers and 5 choices per layer yields approximately $10^{15}$ possible architectures. Early NAS methods relied on Reinforcement Learning to sample thousands of these, which was computationally prohibitive. While recent differentiable NAS methods improved efficiency by building a "supernet" containing all possible paths, they remain memory-intensive and require complex bi-level optimization.
Methodology: The Single-Path Search Space
The core innovation of this work is the Single-path search space. Unlike previous differentiable NAS methods that create a "multi-path" supernet where each candidate operation (e.g., a 3x3 kernel vs. a 5x5 kernel) is a separate branch, Single-Path NAS uses a single-path over-parameterized ConvNet.
As shown in Figure 1, the proposed method encodes all architectural decisions with shared convolutional kernel parameters. The researchers observed that a 3x3 convolution is essentially the "inner core" of a 5x5 convolution. By using a single 5x5 kernel and an indicator function to decide whether to use the "outer shell," they can encode the choice of kernel size directly into the weights of a single path. This "superkernel" approach drastically reduces the number of trainable parameters and the search cost.
Architecture and Optimization
The search space is built upon a fixed macro-architecture inspired by MobileNetV2, consisting of multiple blocks of Mobile Inverted Bottleneck (MBConv) layers. Within each block, the method searches for the optimal kernel size and expansion ratio using the searchable superkernel.
Key technical components include:
- Trainable Indicator Functions: Group Lasso terms are used to determine the importance of different weight subsets, such as the outer shell of a kernel or additional channels for expansion ratios.
- Sigmoid Relaxation: To make discrete architectural choices searchable via gradient descent, the indicator functions are relaxed into continuous sigmoid functions during the search phase.
- Hardware-Aware Loss: The objective function combines standard cross-entropy loss with a differentiable latency term, based on a runtime model profiled on a Google Pixel 1.
Key Results
Single-Path NAS demonstrates exceptional efficiency and performance:
- Efficiency: The search process takes only 8 epochs, totaling approximately 3.75 hours on a TPUv2. This is 5,000x faster than MnasNet and 25x faster than ProxylessNAS.
- Accuracy: On the ImageNet dataset, the resulting model achieved 74.96% top-1 accuracy.
- Hardware Performance: The model runs in 79.48ms on a Pixel 1, outperforming other NAS-designed models (like FBNet and MnasNet) in the same latency bracket by 0.3% to 0.8% accuracy.
Conclusion
Single-Path NAS represents a significant leap in NAS efficiency by reimagining the search space as a single path of shared weights. While the current implementation focuses on searching for kernel sizes and expansion ratios within a fixed backbone, it provides a powerful framework for rapid hardware-efficient design. To support further research, the authors have open-sourced their entire codebase.
Summary updated