Abstract
Neural architecture search (NAS) has a great impact by automatically designing effective neural network architectures. However, the prohibitive computational demand of conventional NAS algorithms (e.g. $10^4$ GPU hours) makes it difficult to directly search the architectures on large-scale tasks (e.g. ImageNet). Differentiable NAS can reduce the cost of GPU hours via a continuous representation of network architecture but suffers from the high GPU memory consumption issue (grow linearly w.r.t. candidate set size). As a result, they need to utilize proxy tasks, such as training on a smaller dataset, or learning with only a few blocks, or training just for a few epochs. These architectures optimized on proxy tasks are not guaranteed to be optimal on the target task. In this paper, we present ProxylessNAS that can directly learn the architectures for large-scale target tasks and target hardware platforms. We address the high memory consumption issue of differentiable NAS and reduce the computational cost (GPU hours and GPU memory) to the same level of regular training while still allowing a large candidate set. Experiments on CIFAR-10 and ImageNet demonstrate the effectiveness of directness and specialization. On CIFAR-10, our model achieves 2.08% test error with only 5.7M parameters, better than the previous state-of-the-art architecture AmoebaNet-B, while using 6x fewer parameters. On ImageNet, our model achieves 3.1% better top-1 accuracy than MobileNetV2, while being 1.2x faster with measured GPU latency. We also apply ProxylessNAS to specialize neural architectures for hardware with direct hardware metrics (e.g. latency) and provide insights for efficient CNN architecture design.
Introduction: The Problem with Proxy Tasks
Neural Architecture Search (NAS) has revolutionized the automation of neural network design, but its resource intensity remains a significant barrier. Traditional NAS methods often require training thousands of candidate models, consuming tens of thousands of GPU hours. To mitigate this, researchers have historically relied on proxy tasks: searching for small architectural building blocks on reduced datasets (like CIFAR-10) and then stacking these blocks to form a larger network for the target task (like ImageNet).
However, this proxy-based approach introduces two major sub-optimalities. First, architectures optimized for small datasets or limited training cycles are rarely optimal for large-scale target tasks. Second, the requirement to repeat the same block structure throughout the network restricts architectural diversity. While differentiable NAS methods (e.g., DARTS) have improved search efficiency, they suffer from "memory explosion" because they must load all candidate operations into GPU memory simultaneously.
Methodology: ProxylessNAS and Path-Level Binarization
ProxylessNAS addresses these bottlenecks by framing architecture search as a large-scale network pruning problem. The search space is defined as an over-parameterized "super-network" containing all candidate operations—such as various convolution sizes, pooling, and identity mappings—as parallel paths.
To overcome the memory constraints of differentiable NAS, the authors introduce path-level binarization. Instead of maintaining a weighted sum of all possible paths in memory, ProxylessNAS uses binary gates to ensure that only a single path is active during any given forward/backward pass. This reduces the memory footprint to the same level as training a standard, compact model, enabling direct search on large-scale datasets like ImageNet.
The training process alternates between two phases:
- Weight Updates: Standard network weights are updated via gradient descent on the sampled active path.
- Architecture Updates: Weights are frozen, and architecture parameters (probabilities for each path) are updated using either a modified gradient-based approach (BinaryConnect) or reinforcement learning (REINFORCE).
Furthermore, ProxylessNAS is hardware-aware. It incorporates latency directly into the loss function. Since latency is non-differentiable, the authors utilize a continuous latency prediction model, trained on measured on-device data (e.g., from a Google Pixel 1), to act as a regularization penalty during the search.
Key Results and Hardware Specialization
The "proxy-less" nature of this approach yields substantial gains in both efficiency and performance:
- Efficiency: ProxylessNAS completed an ImageNet search in just 200 GPU hours, a 200x reduction compared to Google’s MnasNet (~40,000 GPU hours).
- ImageNet Performance: The resulting model achieved 75.1% top-1 accuracy. It matched MobileNetV2's accuracy while being 1.83x faster, or provided a 3.1% accuracy boost at comparable latency.
- CIFAR-10 Performance: The model achieved a 2.08% test error with 5.7M parameters, outperforming AmoebaNet-B while using 6x fewer parameters.
A critical insight from this work is that hardware dictates architecture. The search process generated distinct, specialized models for different platforms:
- GPUs favored shallow, wide networks with large convolutions to exploit parallelism.
- CPUs favored deeper, narrower networks with smaller convolutions.
Conclusion
ProxylessNAS bridges the gap between architecture search and model compression, demonstrating that direct search on target tasks and hardware is not only feasible but superior to proxy-based methods. By eliminating the memory bottleneck through path-level binarization, it allows for the creation of highly specialized, efficient models at a fraction of the traditional computational cost.
Summary updated