BinaryViT: Towards Efficient and Accurate Binary Vision Transformers

Zhikai LiarXiv 2023 · DOI · arXiv

TL;DR

Vision Transformers (ViTs) are powerful but computationally expensive. This paper introduces BinaryViT, a novel method for fully binarizing ViTs (reducing weights and activations to ±1), achieving significant model size and computational savings while maintaining competitive accuracy. BinaryViT tackles the key challenges of weight oscillation and activation distortion during binarization with new gradient regularization and activation shift techniques.

What Problem Does This Paper Address?

Vision Transformers (ViTs) have revolutionized computer vision, but their substantial computational and memory demands make them impractical for resource-constrained devices like mobile phones. Model compression techniques are crucial for deploying ViTs in real-world applications. While binarization (quantizing weights and activations to ±1) offers extreme compression and efficiency gains, it has historically led to significant performance degradation in ViTs, unlike in Convolutional Neural Networks (CNNs). This paper aims to bridge this gap by developing an accurate and efficient full binarization method for ViTs.

Methods

The authors first establish a strong baseline for binarized ViTs by incorporating standard techniques. Through empirical analysis, they identify two primary causes for performance degradation: weight oscillation during training and information distortion in activations.

To address weight oscillation, they propose a Gradient Regularization Scheme (GRS). GRS encourages a bimodal distribution of latent weights, pushing them towards ±1 and away from zero, thereby reducing stochasticity and gradient inversion issues. This is achieved through a novel, smooth, and differentiable regularization term.

To combat activation distortion, they introduce an Activation Shift Module (ASM). ASM adaptively tunes the distribution of activations before the sign function is applied. It learns dynamic shift factors for each token and channel, allowing the binarized model to better preserve semantic information and mitigate the severe inter-channel and inter-token variations observed in binarized activations.

The training process also involves a two-stage binarization approach and knowledge distillation to further improve performance.

Datasets

The primary dataset used for training and evaluation is ImageNet [35]. The authors conduct experiments on this dataset using various ViT architectures.

Key Insights and Ideas

  • Binarization's Unique Challenges in ViTs: Unlike CNNs, ViTs' complex architectures and feature representations present unique hurdles for binarization, leading to more pronounced performance drops.
  • Weight Oscillation is a Key Bottleneck: The paper highlights that the straight-through estimator (STE) used in binarization training can cause latent weights to cluster around zero, leading to unstable training and performance degradation.
  • Activation Distortion is Non-Trivial: Drastic inter-channel and inter-token variations in activations, even after binarization, cause significant information loss. Simple uniform binarization is insufficient.
  • GRS for Stable Weight Training: The proposed GRS effectively regularizes weights into a bimodal distribution, directly addressing the oscillation problem and improving training stability.
  • ASM for Preserving Activation Information: The ASM's adaptive shifting mechanism proves crucial for compensating for activation distortion, allowing the binarized model to retain more meaningful information.

Key Results

BinaryViT achieves significant improvements in efficiency and accuracy:

  • Accuracy: On the ImageNet dataset, BinaryViT consistently surpasses strong baselines. For example, it improves the accuracy of a strong baseline for DeiT-S by 2.05%, bringing the Top-1 accuracy to 65.54%.
  • Compression: BinaryViT offers substantial compression. Compared to the full-precision DeiT-S, it achieves savings of 16.2× in model size and 17.7× in OPs (Operations).
  • Generalization: The method demonstrates effectiveness across various ViT architectures, including DeiT and ViT.

Limitations and Future Work

  • Limitations: The paper notes that while the proposed methods significantly improve binarized ViT performance, there is still a gap compared to full-precision models. The authors also mention that feature distillation was found to be unsuitable for ViTs in their experiments, relying solely on logits distillation.
  • Future Work: The paper suggests that their contributions provide a promising path for compressing ViTs and improving their inference efficiency for real-world applications. Further research could explore more advanced distillation techniques or investigate the application of BinaryViT to other transformer-based architectures and tasks.

Summary updated

Abstract

Vision Transformers (ViTs) have emerged as the fundamental architecture for most computer vision fields, but the considerable memory and computation costs hinders their application on resource-limited devices. As one of the most powerful compression methods, binarization reduces the computation of the neural network by quantizing the weights and activation values as $\pm$1. Although existing binarization methods have demonstrated excellent performance on Convolutional Neural Networks (CNNs), the full binarization of ViTs is still under-studied and suffering a significant performance drop. In this paper, we first argue empirically that the severe performance degradation is mainly caused by the weight oscillation in the binarization training and the information distortion in the activation of ViTs. Based on these analyses, we propose $\textbf{BinaryViT}$, an accurate full binarization scheme for ViTs, which pushes the quantization of ViTs to the limit. Specifically, we propose a novel gradient regularization scheme (GRS) for driving a bimodal distribution of the weights to reduce oscillation in binarization training. Moreover, we design an activation shift module (ASM) to adaptively tune the activation distribution to reduce the information distortion caused by binarization. Extensive experiments on ImageNet dataset show that our BinaryViT consistently surpasses the strong baseline by 2.05% and improve the accuracy of fully binarized ViTs to a usable level. Furthermore, our method achieves impressive savings of 16.2$\times$ and 17.7$\times$ in model size and OPs compared to the full-precision DeiT-S.
BibTeX
@misc{li2023binaryvit,
  author = {Zhikai Li},
  title = {{BinaryViT: Towards Efficient and Accurate Binary Vision Transformers}},
  year = {2023},
  eprint = {2305.14730},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2305.14730},
}
Figure