TL;DR
Vision Transformers (ViTs) are powerful but computationally expensive. This paper introduces BinaryViT, a novel method for fully binarizing ViTs (reducing weights and activations to ±1), achieving significant model size and computational savings while maintaining competitive accuracy. BinaryViT tackles the key challenges of weight oscillation and activation distortion during binarization with new gradient regularization and activation shift techniques.
What Problem Does This Paper Address?
Vision Transformers (ViTs) have revolutionized computer vision, but their substantial computational and memory demands make them impractical for resource-constrained devices like mobile phones. Model compression techniques are crucial for deploying ViTs in real-world applications. While binarization (quantizing weights and activations to ±1) offers extreme compression and efficiency gains, it has historically led to significant performance degradation in ViTs, unlike in Convolutional Neural Networks (CNNs). This paper aims to bridge this gap by developing an accurate and efficient full binarization method for ViTs.
Methods
The authors first establish a strong baseline for binarized ViTs by incorporating standard techniques. Through empirical analysis, they identify two primary causes for performance degradation: weight oscillation during training and information distortion in activations.
To address weight oscillation, they propose a Gradient Regularization Scheme (GRS). GRS encourages a bimodal distribution of latent weights, pushing them towards ±1 and away from zero, thereby reducing stochasticity and gradient inversion issues. This is achieved through a novel, smooth, and differentiable regularization term.
To combat activation distortion, they introduce an Activation Shift Module (ASM). ASM adaptively tunes the distribution of activations before the sign function is applied. It learns dynamic shift factors for each token and channel, allowing the binarized model to better preserve semantic information and mitigate the severe inter-channel and inter-token variations observed in binarized activations.
The training process also involves a two-stage binarization approach and knowledge distillation to further improve performance.
Datasets
The primary dataset used for training and evaluation is ImageNet [35]. The authors conduct experiments on this dataset using various ViT architectures.
Key Insights and Ideas
- Binarization's Unique Challenges in ViTs: Unlike CNNs, ViTs' complex architectures and feature representations present unique hurdles for binarization, leading to more pronounced performance drops.
- Weight Oscillation is a Key Bottleneck: The paper highlights that the straight-through estimator (STE) used in binarization training can cause latent weights to cluster around zero, leading to unstable training and performance degradation.
- Activation Distortion is Non-Trivial: Drastic inter-channel and inter-token variations in activations, even after binarization, cause significant information loss. Simple uniform binarization is insufficient.
- GRS for Stable Weight Training: The proposed GRS effectively regularizes weights into a bimodal distribution, directly addressing the oscillation problem and improving training stability.
- ASM for Preserving Activation Information: The ASM's adaptive shifting mechanism proves crucial for compensating for activation distortion, allowing the binarized model to retain more meaningful information.
Key Results
BinaryViT achieves significant improvements in efficiency and accuracy:
- Accuracy: On the ImageNet dataset, BinaryViT consistently surpasses strong baselines. For example, it improves the accuracy of a strong baseline for DeiT-S by 2.05%, bringing the Top-1 accuracy to 65.54%.
- Compression: BinaryViT offers substantial compression. Compared to the full-precision DeiT-S, it achieves savings of 16.2× in model size and 17.7× in OPs (Operations).
- Generalization: The method demonstrates effectiveness across various ViT architectures, including DeiT and ViT.
Limitations and Future Work
- Limitations: The paper notes that while the proposed methods significantly improve binarized ViT performance, there is still a gap compared to full-precision models. The authors also mention that feature distillation was found to be unsuitable for ViTs in their experiments, relying solely on logits distillation.
- Future Work: The paper suggests that their contributions provide a promising path for compressing ViTs and improving their inference efficiency for real-world applications. Further research could explore more advanced distillation techniques or investigate the application of BinaryViT to other transformer-based architectures and tasks.
Summary updated