BViT: Broad Attention based Vision Transformer

Nannan Li, Yaran Chen, Weifan Li, Zixiang Ding, Dongbin Zhao

Introduction

Vision Transformers (ViTs) have revolutionized computer vision by applying self-attention to image patches. However, standard architectures typically process layers in isolation, stacking them to form a deep network. This hierarchical approach often leads to significant redundancy, where different layers focus on similar features, and a potential loss of fine-grained local information in deeper stages.

BViT (Broad Attention based Vision Transformer) addresses these limitations by introducing a "broad attention" mechanism. This approach integrates attention information across the entire depth of the network, allowing the model to leverage complementary features from different layers without adding new trainable parameters.

The Broad Attention Mechanism

The core innovation of BViT is the Broad Attention Block, which breaks the silos between transformer layers through two primary components:

  1. Broad Connection: Instead of treating each layer's Query ($Q$), Key ($K$), and Value ($V$) matrices as independent entities, BViT concatenates these matrices from all preceding layers. This creates a global repository of attention information that spans the backbone's depth.
  2. Parameter-free Attention: To process this aggregated information efficiently, the authors employ a parameter-free self-attention mechanism. It performs the standard calculation—$softmax(QK^T/\sqrt{d})V$—directly on the concatenated matrices. By reusing existing features, BViT enhances the model's representational power with zero additional trainable parameters.
Figure 2: The overall architecture of BViT, illustrating the integration of Broad Attention with the standard Transformer backbone.

The final output is a weighted combination of the deep features from the final layer and the newly generated "broad features." This "plug-and-play" design allows broad attention to be easily integrated into various existing architectures like Swin Transformer or T2T-ViT.

Key Insights

  • Reduced Redundancy: Analysis using Centered Kernel Alignment (CKA) similarity scores shows that BViT layers learn more distinct, non-redundant features compared to vanilla ViTs.
  • Enhanced Locality: Visualization of attention distances reveals that BViT maintains better focus on local information in shallower layers. This improved "locality" helps the model generalize better, especially on smaller datasets where global-only focus can lead to overfitting.
  • Improved Gradient Flow: The broad connections provide more direct paths for gradients to flow from the loss function to earlier layers, facilitating more effective training of deep architectures.

Experimental Results

BViT was evaluated on major benchmarks including ImageNet-1K, CIFAR-10, and CIFAR-100. The results consistently demonstrate superior performance and efficiency.

Figure 1: Accuracy vs. Parameter count comparison. BViT (ours) consistently outperforms established models like Swin, DeiT, and ResNet across different scales.
  • ImageNet Performance: BViT-5M achieved a top-1 accuracy of 75.0%, while the larger BViT-22M reached 81.6%. Notably, BViT-22M outperformed the popular Swin-T and ResNet-50 while maintaining a smaller parameter footprint.
  • Versatility: When integrated into other models like Swin Transformer and LVT, broad attention provided a consistent performance boost of over 1% on ImageNet without increasing the parameter count.
  • Transfer Learning: The model showed excellent generalization on smaller datasets, reaching 98.9% on CIFAR-10 and 89.9% on CIFAR-100.

Conclusion and Future Work

BViT demonstrates that cross-layer communication via parameter-free attention can significantly boost the performance of Vision Transformers. While the approach adds a negligible increase in FLOPs (approximately $10^{-5}G$), it offers a "free" accuracy improvement in terms of model size. Future research may involve using Neural Architecture Search (NAS) to optimize layer connections further, potentially finding even more efficient ways to balance performance and computational cost.

Summary updated

Abstract

Recent works have demonstrated that transformer can achieve promising performance in computer vision, by exploiting the relationship among image patches with self-attention. They only consider the attention in a single feature layer, but ignore the complementarity of attention in different layers. In this article, we propose broad attention to improve the performance by incorporating the attention relationship of different layers for vision transformer (ViT), which is called BViT. The broad attention is implemented by broad connection and parameter-free attention. Broad connection of each transformer layer promotes the transmission and integration of information for BViT. Without introducing additional trainable parameters, parameter-free attention jointly focuses on the already available attention information in different layers for extracting useful information and building their relationship. Experiments on image classification tasks demonstrate that BViT delivers superior accuracy of 75.0%/81.6% top-1 accuracy on ImageNet with 5M/22M parameters. Moreover, we transfer BViT to downstream object recognition benchmarks to achieve 98.9% and 89.9% on CIFAR10 and CIFAR100, respectively, that exceed ViT with fewer parameters. For the generalization test, the broad attention in Swin Transformer, T2T-ViT and LVT also brings an improvement of more than 1%. To sum up, broad attention is promising to promote the performance of attention-based models. Code and pretrained models are available at https://github.com/DRL/BViT .
BibTeX
@article{li2024bvit,
  author = {Nannan Li and Yaran Chen and Weifan Li and Zixiang Ding and Dongbin Zhao},
  title = {{BViT: Broad Attention based Vision Transformer}},
  journal = {IEEE Trans. Neural Networks Learn. Syst.},
  year = {2024},
  url = {https://openreview.net/forum?id=SzfwA0nxWL},
}
Figure