Vision Transformers (ViTs) have revolutionized large-scale computer vision, but they often struggle to compete with highly optimized Convolutional Neural Networks (CNNs) in the "mobile" or efficient compute regime (under 1G FLOPs).
To find efficient architectures, researchers typically rely on Supernet-based One-Shot Neural Architecture Search (NAS). This involves training a massive "supernet" that contains all possible architectural paths, and then sampling smaller "sub-networks" from it for deployment. However, directly applying this standard NAS approach to ViTs leads to disastrously poor performance—often worse than simply training a single model from scratch.
The NASViT paper identifies the root cause of this failure and introduces a novel NAS framework specifically designed to overcome it, discovering a family of hybrid models that outperform state-of-the-art CNNs and ViTs in the 200M–800M FLOPs range.
The Culprit: Gradient Conflict
The primary reason supernet training fails for ViTs is a phenomenon called Gradient Conflict. In a supernet, the mathematical directions (gradients) required to improve the giant supernet often point in the opposite direction of those needed to improve the smaller, sampled sub-networks.
The authors quantified this issue, finding that ViTs suffer from a "gradient conflict ratio" of ~34%, which is significantly higher than the ~28% observed in CNN-based NAS (like AlphaNet). This tug-of-war leads to early training saturation and mediocre final models, explaining why standard NAS recipes fail for Transformers.
NASViT's Solutions
To harmonize supernet training and prioritize the performance of the deployable sub-networks, NASViT proposes three key techniques:
- Gradient Projection: When the gradient of the supernet conflicts with a sub-network (indicated by a negative cosine similarity), the algorithm "projects" the supernet gradient so that it becomes orthogonal to the sub-network's gradient. This ensures that updating the supernet never actively harms the progress of the sub-networks.
- Switchable Scaling Layers: The authors introduce channel-wise scaling layers (similar to LayerScale) but make them "switchable." Different sub-network configurations use independent scaling parameters, providing more optimization flexibility and reducing interference between different model sizes.
- Simplified Training Recipe: Counter-intuitively, the heavy data augmentation and regularization (like Mixup or CutMix) typically used to train single ViTs actually exacerbate gradient conflicts in a supernet. NASViT uses a much "weaker" recipe—minimal RandAugment and zero weight decay/dropout during the supernet phase—to provide a clearer training signal.
These techniques are applied to a hybrid architecture search space inspired by LeViT. As shown below, it combines efficient CNN blocks (MBConv) for high-resolution processing with Transformer blocks for capturing global context at lower resolutions.
Key Insights
- Sub-networks First: The core philosophy of NASViT is to prioritize the sub-networks. Since the end goal is to deploy an efficient model, the supernet is treated merely as a "teacher" or container; its own optimization is secondary to the health of its constituents.
- Less is More in Supernets: While single ViTs need heavy regularization to avoid overfitting, supernets are naturally regularized by the weight-sharing process itself. Adding more noise via data augmentation just makes the gradient conflict worse.
- Small Heads Matter: The authors found that using a smaller attention head dimension (e.g., 8 instead of the standard 32 or 64) yields better performance in the efficient regime.
Results and Performance
The resulting NASViT models establish a new state-of-the-art for efficient vision architectures.
- ImageNet Superiority: NASViT models achieve 78.2% to 81.8% top-1 accuracy within the 200M to 800M FLOPs range. For instance, NASViT-A4 (591M FLOPs) achieves 81.4% accuracy, outperforming both the CNN-based AlphaNet (81.1%) and the Transformer-based LeViT (78.6%) at similar or higher compute costs.
- Dominating the Pareto Front: As illustrated below, NASViT models show a significantly better Pareto front for FLOPs-vs-Accuracy compared to prior art.
- Downstream Transfer: When used as a backbone for semantic segmentation, NASViT achieves 73.2% mIoU on Cityscapes and 37.9% on ADE20K with only 5G FLOPs, outperforming previous SOTA backbones.
Limitations
While highly effective, the approach has a few limitations. The training process relies on a strong external teacher (EfficientNet-B5), meaning performance is somewhat tethered to the quality of this teacher. Additionally, the search space is heavily influenced by the LeViT design; exploring more radical architectural departures remains an area for future work. Finally, while "one-shot" NAS is faster than training thousands of models from scratch, the initial supernet training with gradient projection still requires significant GPU resources.
Summary updated