MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer

Sachin Mehta, Mohammad Rastegari

Abstract

Light-weight convolutional neural networks (CNNs) are the de-facto for mobile vision tasks. Their spatial inductive biases allow them to learn representations with fewer parameters across different vision tasks. However, these networks are spatially local. To learn global representations, self-attention-based vision transformers (ViTs) have been adopted. Unlike CNNs, ViTs are heavy-weight. In this paper, we ask the following question: is it possible to combine the strengths of CNNs and ViTs to build a light-weight and low latency network for mobile vision tasks? Towards this end, we introduce MobileViT, a light-weight and general-purpose vision transformer for mobile devices. MobileViT presents a different perspective for the global processing of information with transformers, i.e., transformers as convolutions. Our results show that MobileViT significantly outperforms CNN- and ViT-based networks across different tasks and datasets. On the ImageNet-1k dataset, MobileViT achieves top-1 accuracy of 78.4% with about 6 million parameters, which is 3.2% and 6.2% more accurate than MobileNetv3 (CNN-based) and DeIT (ViT-based) for a similar number of parameters. On the MS-COCO object detection task, MobileViT is 5.7% more accurate than MobileNetv3 for a similar number of parameters.

Introduction: The Local vs. Global Dilemma

For years, Convolutional Neural Networks (CNNs) like MobileNet have been the gold standard for mobile vision because of their spatial inductive bias—the inherent assumption that pixels near each other are related. This makes them parameter-efficient and easy to train. However, CNNs are "spatially local"; they struggle to capture long-range dependencies in an image without becoming very deep and computationally expensive.

Vision Transformers (ViTs) solved this by using self-attention to capture global context, but they come with a "heavyweight" price tag. They lack the inductive bias of CNNs, meaning they require massive datasets, complex data augmentation, and many more parameters to learn effectively. When scaled down to mobile sizes, standard ViTs perform significantly worse than their CNN counterparts. MobileViT asks: Can we get the global reach of a Transformer with the efficiency of a CNN?

Methodology: Transformers as Convolutions

The core innovation is the MobileViT block, which aims to model both local and global information in a tensor with very few parameters.

Comparison of standard ViT and MobileViT architectures.
  1. Local Processing: The block starts with standard $n \times n$ and $1 \times 1$ convolutions to encode local spatial information and project features into a higher-dimensional space.
  2. Transformers as Convolutions: Instead of flattening the entire image into a long sequence (which is computationally expensive), MobileViT "unfolds" the feature map into patches. It then applies a Transformer to these patches such that each pixel attends to every other pixel at the same relative location across all patches. This gives every pixel a global receptive field of the entire image ($H \times W$).
  3. Folding and Fusion: The patches are "folded" back into a feature map, projected back to the original dimension, and concatenated with the original input (a skip-connection) to preserve local features.

Additionally, the authors introduced a Multi-scale Sampler for training. Unlike standard training that uses a fixed resolution (e.g., $224 \times 224$), this sampler varies the image size and batch size dynamically during training. This not only speeds up training but also helps the model generalize better across different resolutions.

Key Results and Performance

MobileViT consistently outperformed both CNN-based and ViT-based models of similar sizes across multiple benchmarks:

  • ImageNet-1k: At a budget of ~6 million parameters, MobileViT achieved 78.4% top-1 accuracy. This is 3.2% higher than MobileNetv3 (CNN) and 6.2% higher than DeIT (ViT).
  • Object Detection: On MS-COCO, MobileViT was 5.7% more accurate than MobileNetv3 for a similar parameter count.
  • Semantic Segmentation: When used as a backbone for DeepLabv3, it was 1.4% more accurate while being 1.6x smaller than a MobileNetv2-based version.
MobileViT performance comparison on ImageNet and MS-COCO.

On an iPhone 12, MobileViT models run in real-time (approx. 7ms to 12ms for classification and detection). While they are currently slightly slower than highly optimized CNNs due to a lack of dedicated hardware kernels for Transformers on mobile chips, their parameter efficiency is unmatched.

Discussion and Conclusion

The primary limitation is inference latency on current hardware. While MobileViT has fewer FLOPs (floating-point operations) than many CNNs, it is currently slower on mobile devices because mobile chips are heavily optimized for standard convolutions (via fused BatchNorm, etc.). Transformers require "unfolding" and "folding" operations that don't yet have highly optimized, dedicated hardware kernels.

By retaining the CNN structure, MobileViT doesn't have to "re-learn" how images work from scratch, allowing it to reach higher accuracies with millions fewer parameters. As mobile hardware and software libraries (like CoreML or TFLite) evolve to better support Transformer operations, the real-world speed of MobileViT will likely surpass that of traditional CNNs.

Summary updated

Abstract

Light-weight convolutional neural networks (CNNs) are the de-facto for mobile vision tasks. Their spatial inductive biases allow them to learn representations with fewer parameters across different vision tasks. However, these networks are spatially local. To learn global representations, self-attention-based vision trans-formers (ViTs) have been adopted. Unlike CNNs, ViTs are heavy-weight. In this paper, we ask the following question: is it possible to combine the strengths of CNNs and ViTs to build a light-weight and low latency network for mobile vision tasks? Towards this end, we introduce MobileViT, a light-weight and general-purpose vision transformer for mobile devices. MobileViT presents a different perspective for the global processing of information with transformers, i.e., transformers as convolutions. Our results show that MobileViT significantly outperforms CNN- and ViT-based networks across different tasks and datasets. On the ImageNet-1k dataset, MobileViT achieves top-1 accuracy of 78.4% with about 6 million parameters, which is 3.2% and 6.2% more accurate than MobileNetv3 (CNN-based) and DeIT (ViT-based) for a similar number of parameters. On the MS-COCO object detection task, MobileViT is 5.7% more accurate than MobileNetv3 for a similar number of parameters. Our source code is open-source and available at: https://github.com/apple/ml-cvnets
BibTeX
@inproceedings{mehta2022mobilevit,
  author = {Sachin Mehta and Mohammad Rastegari},
  title = {{MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=vh-0sUt8HlG},
}
Figure