NASViT: Neural Architecture Search for Efficient Vision Transformers with Gradient Conflict aware Supernet Training

Chengyue Gong, Dilin Wang, Meng Li, Xinlei Chen, Zhicheng Yan, Yuandong Tian, Qiang Liu, Vikas ChandraICLR 2022 · OpenReview

Vision Transformers (ViTs) have revolutionized large-scale computer vision, but they often struggle to compete with highly optimized Convolutional Neural Networks (CNNs) in the "mobile" or efficient compute regime (under 1G FLOPs).

To find efficient architectures, researchers typically rely on Supernet-based One-Shot Neural Architecture Search (NAS). This involves training a massive "supernet" that contains all possible architectural paths, and then sampling smaller "sub-networks" from it for deployment. However, directly applying this standard NAS approach to ViTs leads to disastrously poor performance—often worse than simply training a single model from scratch.

The NASViT paper identifies the root cause of this failure and introduces a novel NAS framework specifically designed to overcome it, discovering a family of hybrid models that outperform state-of-the-art CNNs and ViTs in the 200M–800M FLOPs range.

The Culprit: Gradient Conflict

The primary reason supernet training fails for ViTs is a phenomenon called Gradient Conflict. In a supernet, the mathematical directions (gradients) required to improve the giant supernet often point in the opposite direction of those needed to improve the smaller, sampled sub-networks.

The authors quantified this issue, finding that ViTs suffer from a "gradient conflict ratio" of ~34%, which is significantly higher than the ~28% observed in CNN-based NAS (like AlphaNet). This tug-of-war leads to early training saturation and mediocre final models, explaining why standard NAS recipes fail for Transformers.

NASViT's Solutions

To harmonize supernet training and prioritize the performance of the deployable sub-networks, NASViT proposes three key techniques:

  1. Gradient Projection: When the gradient of the supernet conflicts with a sub-network (indicated by a negative cosine similarity), the algorithm "projects" the supernet gradient so that it becomes orthogonal to the sub-network's gradient. This ensures that updating the supernet never actively harms the progress of the sub-networks.
  2. Switchable Scaling Layers: The authors introduce channel-wise scaling layers (similar to LayerScale) but make them "switchable." Different sub-network configurations use independent scaling parameters, providing more optimization flexibility and reducing interference between different model sizes.
  3. Simplified Training Recipe: Counter-intuitively, the heavy data augmentation and regularization (like Mixup or CutMix) typically used to train single ViTs actually exacerbate gradient conflicts in a supernet. NASViT uses a much "weaker" recipe—minimal RandAugment and zero weight decay/dropout during the supernet phase—to provide a clearer training signal.

These techniques are applied to a hybrid architecture search space inspired by LeViT. As shown below, it combines efficient CNN blocks (MBConv) for high-resolution processing with Transformer blocks for capturing global context at lower resolutions.

Hybrid architecture search space combining CNN and Transformer blocks

Key Insights

  • Sub-networks First: The core philosophy of NASViT is to prioritize the sub-networks. Since the end goal is to deploy an efficient model, the supernet is treated merely as a "teacher" or container; its own optimization is secondary to the health of its constituents.
  • Less is More in Supernets: While single ViTs need heavy regularization to avoid overfitting, supernets are naturally regularized by the weight-sharing process itself. Adding more noise via data augmentation just makes the gradient conflict worse.
  • Small Heads Matter: The authors found that using a smaller attention head dimension (e.g., 8 instead of the standard 32 or 64) yields better performance in the efficient regime.

Results and Performance

The resulting NASViT models establish a new state-of-the-art for efficient vision architectures.

  • ImageNet Superiority: NASViT models achieve 78.2% to 81.8% top-1 accuracy within the 200M to 800M FLOPs range. For instance, NASViT-A4 (591M FLOPs) achieves 81.4% accuracy, outperforming both the CNN-based AlphaNet (81.1%) and the Transformer-based LeViT (78.6%) at similar or higher compute costs.
  • Dominating the Pareto Front: As illustrated below, NASViT models show a significantly better Pareto front for FLOPs-vs-Accuracy compared to prior art.
Top-1 validation accuracy vs FLOPs showing NASViT outperforming prior models
  • Downstream Transfer: When used as a backbone for semantic segmentation, NASViT achieves 73.2% mIoU on Cityscapes and 37.9% on ADE20K with only 5G FLOPs, outperforming previous SOTA backbones.

Limitations

While highly effective, the approach has a few limitations. The training process relies on a strong external teacher (EfficientNet-B5), meaning performance is somewhat tethered to the quality of this teacher. Additionally, the search space is heavily influenced by the LeViT design; exploring more radical architectural departures remains an area for future work. Finally, while "one-shot" NAS is faster than training thousands of models from scratch, the initial supernet training with gradient projection still requires significant GPU resources.

Summary updated

Abstract

Designing accurate and efficient vision transformers (ViTs) is a highly important but challenging task. Supernet-based one-shot neural architecture search (NAS) enables fast architecture optimization and has achieved state-of-the-art (SOTA) results on convolutional neural networks (CNNs). However, directly applying the supernet-based NAS to optimize ViTs leads to poor performance - even worse compared to training single ViTs. In this work, we observe that the poor performance is due to a gradient conflict issue: the gradients of different sub-networks conflict with that of the supernet more severely in ViTs than CNNs, which leads to early saturation in training and inferior convergence. To alleviate this issue, we propose a series of techniques, including a gradient projection algorithm, a switchable layer scaling design, and a simplified data augmentation and regularization training recipe. The proposed techniques significantly improve the convergence and the performance of all sub-networks. Our discovered hybrid ViT model family, dubbed NASViT, achieves top-1 accuracy from 78.2% to 81.8% on ImageNet from 200M to 800M FLOPs, and outperforms all the prior art CNNs and ViTs, including AlphaNet and LeViT, etc. When transferred to semantic segmentation tasks, NASViTs also outperform previous backbones on both Cityscape and ADE20K datasets, achieving 73.2% and 37.9% mIoU with only 5G FLOPs, respectively. Code is available at https://github.com/facebookresearch/NASViT.
BibTeX
@inproceedings{gong2022nasvit,
  author = {Chengyue Gong and Dilin Wang and Meng Li and Xinlei Chen and Zhicheng Yan and Yuandong Tian and Qiang Liu and Vikas Chandra},
  title = {{NASViT: Neural Architecture Search for Efficient Vision Transformers with Gradient Conflict aware Supernet Training}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=zzWFu-TNCw},
}
Figure