Abstract
Modern vision models often scale performance by increasing depth or width, leading to significant computational overhead. This paper introduces Gramian Attention (GA) heads, a lightweight architectural enhancement that boosts model capacity without substantial bloat. By initializing class tokens with a Gramian matrix and employing a novel decorrelation loss to ensure feature diversity across multiple heads, GA-networks achieve superior accuracy-throughput trade-offs. Experimental results on ImageNet-1K and downstream tasks demonstrate that GA-networks outperform significantly larger backbones while maintaining high efficiency.
Introduction
The prevailing trend in computer vision is to scale models by adding layers or channels, which increases parameter counts and slows deployment. While multi-head architectures have been explored to capture diverse features, they often suffer from redundancy or require heavy decoder components. This research addresses the challenge of increasing network capacity using lightweight, plug-and-play components that learn unique and useful representations.
Methodology
Gramian Attention
In standard Vision Transformers (ViTs), the class token is a learned vector that interacts with image patches. The proposed Gramian Attention mechanism replaces the random initialization of this token with a Gramian matrix computed from the input features. This matrix captures pairwise similarities and spatial correlations, providing the attention layer with a rich structural prior from the start.
Decorrelation Loss ($L_{dec}$)
To maximize the utility of multiple lightweight heads, the authors introduce a decorrelation loss. Unlike standard knowledge distillation which encourages similarity, this loss acts as "inverse distillation," pushing the feature representations of different heads apart. This forces each head to capture a distinct perspective of the data, preventing the model from collapsing into redundant solutions.
Experimental Results
The GA-network approach was evaluated across several benchmarks, including ImageNet-1K, COCO, ADE20k, and fine-grained classification tasks.
- Efficiency and Accuracy: A GA-ResNet50 achieved 82.5% Top-1 accuracy on ImageNet-1K, surpassing the much larger ResNet152 (81.8%) while maintaining a higher throughput (2145 vs 1463 images/sec).
- Transformer Enhancements: Integrating GA heads into a CSwin-T backbone improved accuracy from 82.7% to 84.1%.
- Downstream Performance: In semantic segmentation on ADE20k, a GA-ResNet50 outperformed a Swin-T backbone while utilizing approximately 45% fewer parameters in the UperNet configuration.
Discussion and Key Insights
The success of GA-networks stems from two primary factors. First, the Gramian matrix effectively performs bilinear pooling within the attention framework, capturing delicate spatial details. Second, the diversity enforced by the decorrelation loss ensures that the aggregate of multiple "weak" heads is more powerful than a single "strong" head.
Furthermore, the benefits extend to the backbone itself; even if GA heads are removed after training, the underlying network remains more robust due to the diverse gradients received during the training process.
Conclusion
Gramian Attention heads provide a highly efficient method for scaling vision models. By focusing on feature correlation and diversity rather than sheer size, GA-networks set a new standard for accuracy-throughput trade-offs in computer vision. Future work will explore the scalability of this approach to extremely large-scale models and specialized backbone designs.
Summary updated