Gramian Attention Heads are Strong yet Efficient Vision Learners

Jongbin Ryu, Dongyoon Han, Jongwoo Lim

Abstract

Modern vision models often scale performance by increasing depth or width, leading to significant computational overhead. This paper introduces Gramian Attention (GA) heads, a lightweight architectural enhancement that boosts model capacity without substantial bloat. By initializing class tokens with a Gramian matrix and employing a novel decorrelation loss to ensure feature diversity across multiple heads, GA-networks achieve superior accuracy-throughput trade-offs. Experimental results on ImageNet-1K and downstream tasks demonstrate that GA-networks outperform significantly larger backbones while maintaining high efficiency.

Introduction

The prevailing trend in computer vision is to scale models by adding layers or channels, which increases parameter counts and slows deployment. While multi-head architectures have been explored to capture diverse features, they often suffer from redundancy or require heavy decoder components. This research addresses the challenge of increasing network capacity using lightweight, plug-and-play components that learn unique and useful representations.

Methodology

Gramian Attention

In standard Vision Transformers (ViTs), the class token is a learned vector that interacts with image patches. The proposed Gramian Attention mechanism replaces the random initialization of this token with a Gramian matrix computed from the input features. This matrix captures pairwise similarities and spatial correlations, providing the attention layer with a rich structural prior from the start.

Decorrelation Loss ($L_{dec}$)

To maximize the utility of multiple lightweight heads, the authors introduce a decorrelation loss. Unlike standard knowledge distillation which encourages similarity, this loss acts as "inverse distillation," pushing the feature representations of different heads apart. This forces each head to capture a distinct perspective of the data, preventing the model from collapsing into redundant solutions.

Experimental Results

The GA-network approach was evaluated across several benchmarks, including ImageNet-1K, COCO, ADE20k, and fine-grained classification tasks.

  • Efficiency and Accuracy: A GA-ResNet50 achieved 82.5% Top-1 accuracy on ImageNet-1K, surpassing the much larger ResNet152 (81.8%) while maintaining a higher throughput (2145 vs 1463 images/sec).
  • Transformer Enhancements: Integrating GA heads into a CSwin-T backbone improved accuracy from 82.7% to 84.1%.
  • Downstream Performance: In semantic segmentation on ADE20k, a GA-ResNet50 outperformed a Swin-T backbone while utilizing approximately 45% fewer parameters in the UperNet configuration.

Discussion and Key Insights

The success of GA-networks stems from two primary factors. First, the Gramian matrix effectively performs bilinear pooling within the attention framework, capturing delicate spatial details. Second, the diversity enforced by the decorrelation loss ensures that the aggregate of multiple "weak" heads is more powerful than a single "strong" head.

Furthermore, the benefits extend to the backbone itself; even if GA heads are removed after training, the underlying network remains more robust due to the diverse gradients received during the training process.

Conclusion

Gramian Attention heads provide a highly efficient method for scaling vision models. By focusing on feature correlation and diversity rather than sheer size, GA-networks set a new standard for accuracy-throughput trade-offs in computer vision. Future work will explore the scalability of this approach to extremely large-scale models and specialized backbone designs.

Summary updated

Abstract

We introduce a novel architecture design that enhances expressiveness by incorporating multiple head classifiers (\ie, classification heads) instead of relying on channel expansion or additional building blocks. Our approach employs attention-based aggregation, utilizing pairwise feature similarity to enhance multiple lightweight heads with minimal resource overhead. We compute the Gramian matrices to reinforce class tokens in an attention layer for each head. This enables the heads to learn more discriminative representations, enhancing their aggregation capabilities. Furthermore, we propose a learning algorithm that encourages heads to complement each other by reducing correlation for aggregation. Our models eventually surpass state-of-the-art CNNs and ViTs regarding the accuracy-throughput trade-off on ImageNet-1K and deliver remarkable performance across various downstream tasks, such as COCO object instance segmentation, ADE20k semantic segmentation, and fine-grained visual classification datasets. The effectiveness of our framework is substantiated by practical experimental results and further underpinned by generalization error bound. We release the code publicly at: https://github.com/Lab-LVM/imagenet-models.
BibTeX
@inproceedings{ryu2023gramian,
  author = {Jongbin Ryu and Dongyoon Han and Jongwoo Lim},
  title = {{Gramian Attention Heads are Strong yet Efficient Vision Learners}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
}
Figure