Introduction & TL;DR
ProxylessNAS introduces a neural architecture search (NAS) algorithm capable of searching directly on large-scale datasets like ImageNet without relying on intermediate "proxy tasks." By utilizing a novel path binarization technique to keep GPU memory costs low, the algorithm designs specialized models that outperform human-designed architectures like MobileNetV2 in both accuracy and hardware-specific latency.
The Problem with Proxy Tasks
Neural Architecture Search (NAS) has successfully automated the design of high-performing deep learning models. However, traditional NAS is prohibitively expensive, often requiring thousands of GPU hours. To make it computationally feasible, researchers typically rely on proxy tasks: they search for a small "building block" or cell on a tiny dataset (like CIFAR-10) and then stack that block repeatedly to build a larger model for a target task (like ImageNet).
The authors identify two major flaws in this conventional "proxy" paradigm:
- Suboptimality: A block that works well on a small dataset like CIFAR-10 is not guaranteed to be optimal for a large-scale dataset like ImageNet.
- Lack of Specialization: Stacking the same block ignores the fact that different stages of a network (early vs. late layers) have distinct computational requirements. Furthermore, it ignores hardware-specific constraints; a model optimized for a GPU might perform poorly on a mobile CPU.
While differentiable NAS (like DARTS) reduced search time, it suffered from memory explosion because it kept all possible candidate operations in GPU memory simultaneously. ProxylessNAS aims to solve this memory bottleneck to enable direct search on the final task and hardware.
ProxylessNAS Methodology
The core innovation of ProxylessNAS is Path Binarization. Instead of holding all candidate operations in memory and weighting them with continuous parameters, the algorithm binarizes the architecture parameters.
- Memory Efficiency: During a training step, only one path (e.g., a $3 \times 3$ convolution) is active. This reduces the memory requirement to the same level as training a single compact model, regardless of how many candidate operations exist in the search space.
- Gradient-Based Search: To make this binarized selection differentiable, the authors use a sampling trick: they sample two paths at a time, update their architecture parameters based on performance, and then rescale the remaining parameters. This allows the model to effectively "prune" less effective paths over time.
- Hardware-Aware Loss: The authors incorporate hardware latency directly into the objective function. They build a latency prediction model that estimates how long an operation takes on a specific device (CPU, GPU, or Mobile). By treating expected latency as a differentiable regularization term, the search can dynamically trade off accuracy for speed.
- No More "Cells": Unlike previous methods that repeat a single searched "cell," ProxylessNAS allows every layer in the network to be unique, providing much higher architectural flexibility.
Evaluation Datasets
The paper evaluates the method across standard benchmarks:
- CIFAR-10: A standard small-scale image classification benchmark.
- ImageNet: The gold-standard large-scale image classification dataset.
- Hardware Platforms: Latency was measured and optimized directly for the Google Pixel 1 (Mobile), NVIDIA V100 (GPU), and Intel Xeon CPU.
Key Insights on Hardware Specialization
The search revealed fascinating insights regarding how architectures should adapt to specific hardware:
- Direct Search Wins: Searching directly on ImageNet yields significantly better results than searching on CIFAR-10 and transferring the architecture.
- GPU vs. CPU Preferences:
- GPU models prefer shallower and wider architectures with early pooling to maximize parallelism.
- CPU models prefer deeper and narrower architectures with late pooling.
- Kernel Sizes: The algorithm found that later layers in a network often benefit from larger kernels (e.g., $7 \times 7$), a pattern rarely utilized in human-designed mobile models.
- NAS as Pruning: The authors successfully frame NAS as a path-level pruning process on an over-parameterized "super-network," creating a conceptual bridge between NAS and model compression.
Performance and Results
- ImageNet Performance: ProxylessNAS achieved 75.1% top-1 accuracy on ImageNet, which is 3.1% higher than MobileNetV2 while being 1.2x faster on a mobile phone.
- CIFAR-10: It reached a 2.08% test error using only 5.7M parameters—outperforming the previous state-of-the-art AmoebaNet-B while using 6x fewer parameters.
- Efficiency: The search cost was reduced to just 200 GPU hours for ImageNet. This is 200x faster than Google’s MnasNet, which required a staggering 40,000 GPU hours.
Limitations and Future Directions
- Latency Modeling Dependency: The success of the hardware-aware search depends heavily on the accuracy of the latency prediction model. If the model fails to capture the nuances of a specific hardware's cache or compiler optimizations, the search results might be skewed.
- Super-network Training: While highly memory-efficient, the method still requires training a very large over-parameterized network, which can be challenging to stabilize during the initial phases.
- Future Work: The authors suggest applying this "proxyless" approach to other complex tasks like object detection and semantic segmentation, where the performance gap between proxy tasks and target tasks is typically even larger.
Summary updated