Abstract
Model binarization has made significant progress in enabling real-time and energy-efficient computation for convolutional neural networks (CNN), offering a potential solution to the deployment challenges faced by Vision Transformers (ViTs) on edge devices. However, due to the structural differences between CNN and Transformer architectures, simply applying binary CNN strategies to the ViT models will lead to a significant performance drop. To tackle this challenge, we propose BHViT, a binarization-friendly hybrid ViT architecture and its full binarization model with the guidance of three important observations. Initially, BHViT utilizes the local information interaction and hierarchical feature aggregation technique from coarse to fine levels to address redundant computations stemming from excessive tokens. Then, a novel module based on shift operations is proposed to enhance the performance of the binary Multilayer Perceptron (MLP) module without significantly increasing computational overhead. In addition, an innovative attention matrix binarization method based on quantization decomposition is proposed to evaluate the token's importance in the binarized attention matrix. Finally, we propose a regularization loss to address the inadequate optimization caused by the incompatibility between the weight oscillation in the binary layers and the Adam Optimizer. Extensive experimental results demonstrate that our proposed algorithm achieves SOTA performance among binary ViT methods.
The Challenge of Binarizing ViTs
Vision Transformers (ViTs) have revolutionized computer vision, but their massive parameter counts and high computational costs make them difficult to deploy on resource-constrained "edge" devices like smartphones or IoT sensors. Model binarization—reducing weights and activations to just 1-bit (1 or -1)—is a powerful way to slash memory usage and speed up inference using bitwise operations. However, simply applying standard binarization techniques used for CNNs to ViTs usually results in a massive performance crash.
The authors identify three main culprits for this drop: vanishing gradients due to non-differentiable functions, information loss in binarized attention matrices, and optimizer incompatibility where the Adam optimizer causes "weight oscillation" in binary layers.
The BHViT Architecture
To address these issues, the authors propose BHViT, a "binarization-friendly" hybrid architecture. Instead of a pure transformer, BHViT adopts a multi-stage approach that combines the strengths of convolutions and attention.
Key Innovations:
- Hybrid Token Mixers: BHViT uses MSGDC (Multi-Scale Grouped Dilated Convolutions) in early stages for efficient local feature handling and MSMHA (Multi-Scale Multi-Head Attention) in later stages for global context.
- Quantization Decomposition (QD): This method decomposes the attention matrix into multiple binary bases, allowing the model to represent token importance more accurately than a single-bit matrix.
- Shift-based MLP: The standard MLP is enhanced with spatial "shift" operations (horizontal, vertical, and mixed), enabling the model to capture spatial information across channels with minimal overhead.
- Regularization Loss ($L_{re}$): An $L_1$-regularization term is introduced to push latent weights toward $\pm 1$, stabilizing training and overcoming the limitations of the Adam optimizer for binary weights.
Experimental Results
BHViT was evaluated on major benchmarks including ImageNet-1K, CIFAR-10, ADE20K, and RS-LVF.
On ImageNet-1K, BHViT-Small achieved 70.1% Top-1 accuracy with 1-bit weights and activations. This represents an 11.5% improvement over previous state-of-the-art binarized ViTs (like BiViT) and even outperforms binarized CNNs like ResNet-34. For road segmentation on the RS-LVF dataset, BHViT achieved an mIoU of 85.1%, significantly outperforming binarized ReActNet (63.6%).
Efficiency tests on an ARM Cortex-A76 CPU showed a latency of 157ms, a dramatic reduction compared to the 863ms required by comparable binarized ViTs, proving BHViT's practical utility for real-time edge deployment.
Conclusion
BHViT successfully bridges the performance gap for binarized Vision Transformers. By combining a hybrid architecture with novel attention binarization and optimization strategies, it enables high-performance computer vision on low-power edge devices. Future work will focus on optimizing software kernels to fully realize the hardware acceleration potential of these hybrid modules.
Summary updated