TL;DR
The Euclidean Enhanced Vision Transformer (EVT) introduces explicit, radially decaying spatial priors into the self-attention mechanism using Euclidean distance, mimicking human visual perception. By pairing this with a flexible, one-dimensional token grouping strategy, EVT achieves linear computational complexity and superior inference speeds. The resulting architecture sets new performance benchmarks across image classification (86.6% top-1 accuracy on ImageNet-1K), object detection, and semantic segmentation.
What Problem Does This Paper Address?
While Vision Transformers (ViTs) have revolutionized computer vision, they suffer from two major flaws: a lack of inherent spatial priors (the inductive bias that nearby pixels are highly related, which comes naturally to CNNs) and quadratic computational complexity ($O(N^2)$) relative to the number of image tokens.
Previous works like the Retentive Network-inspired RMT attempted to solve this by introducing a spatial decay matrix based on Manhattan distance and decomposing attention into horizontal and vertical axes. However, Manhattan distance creates non-smooth, axis-aligned biases that don't match how the human eye perceives the world. Furthermore, decomposing attention into sequential horizontal and vertical steps severely hinders GPU parallelism, leading to sluggish inference speeds. EVT fixes these geometric and architectural inefficiencies.
Methods
EVT introduces two core modifications to the standard attention mechanism:
- Euclidean Self-Attention (EuSA): Instead of using Manhattan distance, EVT modulates attention scores using a 2D spatial decay matrix based on Euclidean distance ($L_2$). As tokens get further apart, their attention weight decays radially. This provides smooth, direction-aware gradients during training and allows the model to capture isotropic spatial harmonics.
- 1D Token Grouping: To achieve linear complexity, EVT abandons complex 2D windowing or decomposed attention. Instead, it flattens the 2D feature map into a 1D sequence and applies a simple one-dimensional grouping strategy. The model alternates between Grouped EuSA (capturing short-range dependencies within local groups) and Dilated EuSA (capturing long-range global dependencies by skipping tokens).
The overall architecture follows a standard hierarchical four-stage design, incorporating Conditional Positional Encoding (CPE) and a Local Context Enhance (LCE) module to further boost local feature extraction.
Datasets
The authors validated EVT across several standard computer vision benchmarks:
- Image Classification: ImageNet-1K (1.28M training images, 50K validation images).
- Object Detection & Instance Segmentation: MS-COCO (118K training, 5K validation).
- Semantic Segmentation: ADE20K (20K training, 2K validation).
- Robustness & Out-of-Distribution (OOD): ImageNet-V2, ImageNet-A, ImageNet-R, and ImageNet-Sketch.
Key Insights and Ideas
- The Geometry of Attention: Human attention decays radially from the center of the visual field. By switching to Euclidean distance, EVT's decay matrix closely mirrors the natural spatial knowledge learned by heavily pretrained, native ViTs. Jensen-Shannon (JS) divergence analysis proved that Euclidean decay matches optimal attention distributions much better than Manhattan decay, stabilizing training and reducing overfitting.
- The Counterintuitive Power of 1D Grouping: Standard wisdom suggests that grouping tokens in 2D windows (like Swin) is essential to preserve spatial structure. EVT proves that if your attention mechanism already possesses a strong 2D spatial prior (via the Euclidean decay matrix), you can safely flatten the image and group tokens in 1D.
- Expanded Modeling Range: Mathematically, 1D grouping expands the effective receptive field coverage rate by a factor of $g$ (the number of groups) compared to 2D grouping. This allows tokens to interact across a longer range in fewer layers, while offering much better flexibility and adaptability when changing input image resolutions.
Key Results
EVT achieves an exceptional accuracy-speed trade-off across all scales:
- ImageNet-1K Classification: EVT-T (15M parameters) achieves 83.0% top-1 accuracy, outperforming RMT-T by 0.6%. The large variant, EVT-L, reaches 86.6% top-1 accuracy at $384\times384$ resolution without requiring any extra training data.
- Throughput & Efficiency: Thanks to the highly parallelizable 1D grouping strategy, EVT-T achieves a throughput of 2142 images/second on a single NVIDIA A100 GPU, significantly faster than RMT-T (1650 imgs/s) and Swin-T (1704 imgs/s).
- Downstream Tasks: On MS-COCO object detection (using Mask R-CNN), EVT-B beats RMT-B by 1.1 $AP^b$. On ADE20K semantic segmentation (using UperNet), EVT-B achieves a 52.7% mIoU, outperforming many backbones that are an order of magnitude larger.
Limitations and Future Work
The primary limitation noted in the paper is that the decay coefficient $\gamma$ within the Euclidean matrix must be manually configured as a hyperparameter for each attention head to capture multi-scale features. Future work could explore adaptively learning these decay coefficients during training to make the spatial prior fully dynamic and data-dependent.
Summary updated