EfficientViT: Lightweight Multi-Scale Attention for High-Resolution Dense Prediction

Han Cai, Junyan Li, Muyan Hu, Chuang Gan, Song HanICCV 2023 · DOI · OpenReview

Abstract

High-resolution dense prediction enables many appealing real-world applications, such as computational photography and autonomous driving. However, the vast computational cost makes deploying state-of-the-art high-resolution dense prediction models on hardware devices difficult. This work presents EfficientViT, a new family of high-resolution vision models with novel lightweight multi-scale attention. Unlike prior high-resolution dense prediction models that rely on heavy self-attention, hardware-inefficient large-kernel convolution, or complicated topology structure, our lightweight multi-scale attention achieves a global receptive field and multi-scale learning with only lightweight and hardware-efficient operations. EfficientViT delivers remarkable performance gains over previous state-of-the-art models with significant speedup on diverse hardware platforms, including mobile CPU, edge GPU, and cloud GPU. Without performance loss on Cityscapes, EfficientViT provides up to 8.8× and 3.8× GPU latency reduction over SegFormer and SegNeXt, respectively. For super-resolution, it provides up to 6.4× speedup over Restormer while providing a 0.11dB gain in PSNR.

The Challenge of High-Resolution Dense Prediction

Dense prediction tasks, such as semantic segmentation, require the model to make a decision for every single pixel. While critical for autonomous driving and medical imaging, these tasks are computationally intensive. The primary bottleneck is the standard Vision Transformer (ViT) architecture. ViTs rely on self-attention mechanisms with quadratic complexity ($O(N^2)$), meaning computational costs quadruple when image resolution doubles. This scaling behavior makes real-time processing on mobile CPUs or edge GPUs nearly impossible. Previous attempts to mitigate this often relied on hardware-unfriendly operations, such as massive 21x21 convolutions, which are poorly supported by many deep learning compilers and hardware backends.

Methods: Lightweight Multi-Scale Attention

EfficientViT introduces a novel Lightweight Multi-Scale Attention (MSA) module designed to provide a global receptive field and multi-scale learning using hardware-efficient operations. The architecture is built on two core innovations:

  1. ReLU-based Global Attention: Standard attention uses the Softmax function, which is difficult to optimize and maintains quadratic complexity. By replacing Softmax with a ReLU-based linear attention, the authors leverage the associative property of matrix multiplication to reduce complexity from quadratic to linear ($O(N)$). This significantly reduces latency on mobile and edge hardware.
  2. Multi-Scale Token Aggregation: To compensate for the potential loss of fine-grained detail in linear attention, the module uses small, efficient depthwise convolutions (3x3 and 5x5) to aggregate nearby tokens before they enter the attention layers. This allows the model to capture features at multiple scales simultaneously.
The EfficientViT module architecture featuring lightweight multi-scale attention and multi-scale token aggregation.

These modules are integrated into a backbone-head architecture, utilizing MBConv (Mobile Inverted Bottleneck) layers for local feature extraction and the new MSA modules for global context.

Key Results and Hardware Versatility

EfficientViT was evaluated across several demanding benchmarks, including Cityscapes and ADE20K for segmentation, and DIV2K for super-resolution. The model consistently outperformed existing leaders like SegFormer and SegNeXt in both speed and accuracy across diverse hardware platforms.

  • Segmentation Speedups: On the Cityscapes dataset, EfficientViT-B1 achieved an 8.8x speedup over SegFormer and a 3.8x speedup over SegNeXt on a GPU while maintaining equivalent mIoU.
  • Super-Resolution Efficiency: For image upscaling, EfficientViT provided a 6.4x speedup over Restormer and improved image quality (PSNR).
  • Classification Performance: On ImageNet-1K, EfficientViT-B3 surpassed EfficientNet-B6 in accuracy while running 3.4x faster on a Jetson Nano GPU.
Performance comparison on Cityscapes and ADE20K across various hardware platforms including mobile CPU, edge GPU, and cloud GPU.

Conclusion

The success of EfficientViT demonstrates that linear-time attention is mandatory for scaling high-resolution vision tasks to edge devices. By replacing hardware-bottleneck operations like Softmax and large-kernel convolutions with ReLU-based linear attention and small-kernel multi-scale aggregation, EfficientViT achieves a superior balance of speed and accuracy. Future work may explore scaling these architectures to larger foundation model sizes and applying the backbone to a wider array of dense prediction tasks like depth estimation.

Summary updated

Abstract

High-resolution dense prediction enables many appealing real-world applications, such as computational photography, autonomous driving, etc. However, the vast computational cost makes deploying state-of-the-art high-resolution dense prediction models on hardware devices difficult. This work presents EfficientViT, a new family of high-resolution vision models with novel lightweight multi-scale attention. Unlike prior high-resolution dense prediction models that rely on heavy self-attention, hardware-inefficient large-kernel convolution, or complicated topology structure to obtain good performances, our lightweight multi-scale attention achieves a global receptive field and multi-scale learning (two critical features for high-resolution dense prediction) with only lightweight and hardware-efficient operations. As such, EfficientViT delivers remarkable performance gains over previous state-of-the-art high-resolution dense prediction models with significant speedup on diverse hardware platforms, including mobile CPU, edge GPU, and cloud GPU. Without performance loss on Cityscapes, our EfficientViT provides up to 8.8× and 3.8× GPU latency reduction over SegFormer and SegNeXt, respectively. For super-resolution, EfficientViT provides up to 6.4× speedup over Restormer while providing 0.11dB gain in PSNR.
BibTeX
@inproceedings{cai2023efficientvit,
  author = {Han Cai and Junyan Li and Muyan Hu and Chuang Gan and Song Han},
  title = {{EfficientViT: Lightweight Multi-Scale Attention for High-Resolution Dense Prediction}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.01587},
}
Figure