Introduction: The Efficiency Bottleneck in Vision Backbones
In computer vision, Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs) offer complementary strengths. CNNs provide efficient local feature extraction through inductive bias, while Transformers excel at capturing long-range global context via Multi-Head Self-Attention (MHSA). However, standard ViTs suffer from quadratic computational complexity relative to image resolution, making them expensive for high-resolution tasks.
Previous hybrid models often apply both operators at the same fine-grained pixel level, leading to redundancy. This paper introduces GLMix, a novel block that decouples local and global processing by operating at different granularities.
Methodology: Global-Local Mixing (GLMix)
The core innovation is the GLMix block, which processes images through two parallel branches:
- Local Branch (Fine-grained): Employs lightweight depth-wise convolutions on a regular pixel grid to capture spatial details and local patterns.
- Global Branch (Coarse-grained): Instead of pixel-to-pixel attention, this branch compresses the image into a fixed set of "semantic slots" (e.g., 64 slots). MHSA is performed only on these slots, ensuring constant global modeling cost regardless of input resolution.
The bridge between these branches consists of two differentiable modules:
- Soft Clustering: Groups fine-grained grid features into coarse semantic slots based on similarity.
- Soft Dispatching: Broadcasts the processed global information from the slots back to the original pixel grid.
Key Results and Benchmarks
The resulting GLNet family was evaluated on ImageNet-1k, COCO 2017, and ADE20k.
Performance on ImageNet-1k
GLNet models demonstrate a superior accuracy-efficiency trade-off. For instance, GLNet-STL achieves 82.5% top-1 accuracy on ImageNet-1k, outperforming Swin-T (81.3%) while maintaining a higher throughput (835.9 vs 755.2 im/s). The largest variant, GLNet-16G, reaches 85.0% top-1 accuracy.
Downstream Tasks
- Object Detection (COCO): GLNet-4G achieves 47.1 mAP, significantly outperforming competitors with similar FLOP counts.
- Semantic Segmentation (ADE20k): It reaches 49.6% mIoU, surpassing models like SG-Former.
Emergent Semantics and Future Directions
A notable finding is that the soft clustering module learns to group pixels into semantically meaningful regions (e.g., parts of an object) without explicit supervision. This suggests potential for weakly-supervised tasks.
Future work includes exploring dynamic slot pruning to adapt the number of semantic slots to image complexity and further optimizing the architecture for specific hardware to maximize real-world throughput.
Summary updated