Abstract
As a compression method that can significantly reduce the cost of calculations and memories, model binarization has been extensively studied in convolutional neural networks. However, the recently popular vision transformer models pose new challenges to such a technique, in which the binarized models suffer from serious performance drops. In this paper, an attention shifting is observed in the binary multi-head self-attention module, which can influence the information fusion between tokens and thus hurts the model performance. From the perspective of information theory, we find a correlation between attention scores and the information quantity, further indicating that a reason for such a phenomenon may be the loss of the information quantity induced by constant moduli of binarized tokens. Finally, we reveal the information quantity hidden in the attention maps of binary vision transformers and propose a simple approach to modify the attention values with look-up information tables so that improve the model performance. Extensive experiments on CIFAR-100/TinyImageNet/ImageNet-1k demonstrate the effectiveness of the proposed information-modified attention on binary vision transformers.
1. Introduction
Model binarization is a critical technique for deploying deep learning models on resource-constrained devices, offering a theoretical 32× reduction in memory and massive speedups via bit-wise XNOR operations. While successful in Convolutional Neural Networks (CNNs), Vision Transformers (ViTs) experience severe performance degradation when binarized.
This paper identifies attention shifting as a primary cause for this drop. In real-valued ViTs, the attention mechanism utilizes both vector direction and magnitude to determine token importance. Binarization forces all token magnitudes to a constant value, causing the attention maps to lose their ability to distinguish high-information regions. Consequently, the model focuses on irrelevant image patches, as visualized in Figure 1.
2. Information-Modified Attention (IMA)
From an information-theoretic perspective, the modulus of a real-valued token vector is positively correlated with its entropy, or "information quantity." Binarization eliminates this variation, stripping the model of its ability to route information based on relative importance. To address this, the researchers propose Information-Modified Attention (IMA).
IMA introduces a small, learnable Information Table for each attention head. Since the token dimension $d$ is fixed, there are only $d+1$ possible Hamming distances between a binary query and key. The table stores $d+1$ learnable modification factors. During inference, the binary attention score (derived from the Hamming distance) acts as an index to look up a factor, which is then used to scale the score. This approach is highly efficient, requiring only a simple look-up and one multiplication per attention score, preserving the hardware benefits of binarization.
3. Experiments and Results
BVT-IMA was evaluated on CIFAR-100, TinyImageNet, and ImageNet-1k across multiple architectures, including DeiT (Full-Attention) and Swin/NesT (Local-Attention) transformers. The method consistently achieves state-of-the-art results for 1-bit ViTs:
- DeiT-Small on ImageNet-1k: Reached 47.98% accuracy, a significant improvement over the 30.73% achieved by the previous BiT baseline.
- DeiT-Base on ImageNet-1k: Achieved 74.06% accuracy, outperforming other recent methods like BiViT (69.60%) and BiBERT (67.50%).
- Swin-Tiny on TinyImageNet: BVT-IMA reached 60.85%, nearly 20% higher than NLP-derived binarization methods like BiBERT.
The authors also introduced a new metric, Consistent Relative Magnitude (CRM), to quantify how well a binary attention map preserves the importance ranking seen in real-valued models. They demonstrate that IMA significantly boosts this consistency, explaining the accuracy gains.
4. Conclusion
BVT-IMA addresses the fundamental issue of information loss in binary vision transformers. By establishing a formal link between token magnitude and information quantity, the authors provide a theoretical basis for the observed attention shifting and a practical, efficient solution via learnable look-up tables. While a gap still exists compared to full-precision models (e.g., ~74% vs ~83% on DeiT-Base), BVT-IMA represents a major step toward making high-performance Transformers viable for edge deployment.
Summary updated