Abstract
For real time applications utilizing Deep Neural Networks (DNNs), it is critical that the models achieve high-accuracy on the target task and low-latency inference on the target computing platform. While Neural Architecture Search (NAS) has been effectively used to develop low-latency networks for image classification, there has been relatively little effort to use NAS to optimize DNN architectures for other vision tasks. In this work, we present what we believe to be the first proxyless hardware-aware search targeted for dense semantic segmentation. With this approach, we advance the state-of-the-art accuracy for latency-optimized networks on the Cityscapes semantic segmentation dataset. Our latency-optimized small SqueezeNAS network achieves 68.02% validation class mIOU with less than 35 ms inference times on the NVIDIA AGX Xavier. Our latency-optimized large SqueezeNAS network achieves 73.62% class mIOU with less than 100 ms inference times. We demonstrate that significant performance gains are possible by utilizing NAS to find networks optimized for both the specific task and inference hardware. We also present detailed analysis comparing our networks to recent state-of-the-art architectures.
Introduction
In the world of deep learning, the standard operating procedure for tasks like semantic segmentation is architecture transfer. Researchers typically take a "backbone" network designed for ImageNet classification, remove the classification head, and add a task-specific decoder.
The authors of SqueezeNAS argue this approach is fundamentally flawed for three reasons:
- Accuracy doesn't always transfer: The best network for classifying a cat isn't necessarily the best for finding the exact boundaries of a road.
- MACs are a poor proxy for speed: Most researchers use "Multiply-Accumulate" (MAC) operations to estimate how fast a model is. However, two models with the same MAC count can have a 10x difference in actual latency depending on how the hardware handles specific layer shapes.
- NAS is perceived as too expensive: Traditional Neural Architecture Search (NAS) can take thousands of GPU days, making it inaccessible for most teams.
SqueezeNAS aims to solve these by providing a "proxyless" search—meaning they search directly on the target task (segmentation) and target hardware (NVIDIA Xavier)—using an efficient search method that takes only a few days.
Methods
The authors utilize a supernetwork-based NAS approach. Instead of training thousands of individual models, they define a single "supernetwork" containing a sequence of "superblocks." Each superblock contains 13 possible candidate operations (different kernel sizes, expansion ratios, and dilations).
Key technical components include:
- Gumbel-Softmax Relaxation: This mathematical trick allows the system to use gradient descent to "choose" the best path through the supernetwork, making the search process incredibly fast.
- Latency Lookup Tables: To make the search hardware-aware, they pre-measured the execution time of every possible block on an NVIDIA Xavier. The search algorithm then uses these real-world numbers to penalize "slow" architectures.
- Resource-Aware Loss: The loss function balances pixel-level accuracy with a resource penalty (either MACs or actual latency), controlled by a hyperparameter ($\alpha$).
The primary dataset for both search and evaluation is Cityscapes, consisting of high-resolution urban street scenes. Architectures are pre-trained on ImageNet and COCO before final fine-tuning on segmentation to improve robustness.
Results and Analysis
SqueezeNAS outperforms several state-of-the-art manually designed and NAS-generated networks. The SqueezeNAS-LAT Small model achieved 68.02% mIOU on Cityscapes with an inference time of less than 35ms on the NVIDIA Xavier. The SqueezeNAS-LAT Large reached 73.62% mIOU in under 100ms.
The "MAC Trap"
A key insight from this work is the "MAC Trap": optimizing for fewer operations doesn't always result in a faster model. As shown in the figures below, "Latency-optimized" (LAT) models often have more MACs than their "MAC-optimized" counterparts but run significantly faster on the actual GPU.
Latency-optimized networks tend to push more computation toward the end of the network. By using higher expansion ratios in later layers where the spatial resolution is lower, the models achieve higher "arithmetic intensity," which GPUs handle more efficiently.
Conclusion
SqueezeNAS demonstrates that significant performance gains are possible by utilizing NAS to find networks optimized for both the specific task and inference hardware. While older NAS methods required thousands of GPU days, SqueezeNAS converged in just 7 to 15 GPU days, making it feasible to re-run the search for new hardware targets. As the diversity of AI hardware grows, using NAS to "compile" architectures for specific chips will become a necessity for real-time applications.
Summary updated