Advancing Vision Transformer with Enhanced Spatial Priors

Qihang Fan, Huaibo Huang, Mingrui Chen, Hongmin Liu, Ran HearXiv 2026 · arXiv

TL;DR

The Euclidean Enhanced Vision Transformer (EVT) introduces explicit, radially decaying spatial priors into the self-attention mechanism using Euclidean distance, mimicking human visual perception. By pairing this with a flexible, one-dimensional token grouping strategy, EVT achieves linear computational complexity and superior inference speeds. The resulting architecture sets new performance benchmarks across image classification (86.6% top-1 accuracy on ImageNet-1K), object detection, and semantic segmentation.

What Problem Does This Paper Address?

While Vision Transformers (ViTs) have revolutionized computer vision, they suffer from two major flaws: a lack of inherent spatial priors (the inductive bias that nearby pixels are highly related, which comes naturally to CNNs) and quadratic computational complexity ($O(N^2)$) relative to the number of image tokens.

Previous works like the Retentive Network-inspired RMT attempted to solve this by introducing a spatial decay matrix based on Manhattan distance and decomposing attention into horizontal and vertical axes. However, Manhattan distance creates non-smooth, axis-aligned biases that don't match how the human eye perceives the world. Furthermore, decomposing attention into sequential horizontal and vertical steps severely hinders GPU parallelism, leading to sluggish inference speeds. EVT fixes these geometric and architectural inefficiencies.

Methods

EVT introduces two core modifications to the standard attention mechanism:

  1. Euclidean Self-Attention (EuSA): Instead of using Manhattan distance, EVT modulates attention scores using a 2D spatial decay matrix based on Euclidean distance ($L_2$). As tokens get further apart, their attention weight decays radially. This provides smooth, direction-aware gradients during training and allows the model to capture isotropic spatial harmonics.
  2. 1D Token Grouping: To achieve linear complexity, EVT abandons complex 2D windowing or decomposed attention. Instead, it flattens the 2D feature map into a 1D sequence and applies a simple one-dimensional grouping strategy. The model alternates between Grouped EuSA (capturing short-range dependencies within local groups) and Dilated EuSA (capturing long-range global dependencies by skipping tokens).
Fig. 2: Overall Architecture of EVT.

The overall architecture follows a standard hierarchical four-stage design, incorporating Conditional Positional Encoding (CPE) and a Local Context Enhance (LCE) module to further boost local feature extraction.

Fig. 3: Illustration of Grouped EuSA (EuSAg) and Dilated EuSA (EuSAd). Different colors represent different groups.

Datasets

The authors validated EVT across several standard computer vision benchmarks:

  • Image Classification: ImageNet-1K (1.28M training images, 50K validation images).
  • Object Detection & Instance Segmentation: MS-COCO (118K training, 5K validation).
  • Semantic Segmentation: ADE20K (20K training, 2K validation).
  • Robustness & Out-of-Distribution (OOD): ImageNet-V2, ImageNet-A, ImageNet-R, and ImageNet-Sketch.

Key Insights and Ideas

  • The Geometry of Attention: Human attention decays radially from the center of the visual field. By switching to Euclidean distance, EVT's decay matrix closely mirrors the natural spatial knowledge learned by heavily pretrained, native ViTs. Jensen-Shannon (JS) divergence analysis proved that Euclidean decay matches optimal attention distributions much better than Manhattan decay, stabilizing training and reducing overfitting.
  • The Counterintuitive Power of 1D Grouping: Standard wisdom suggests that grouping tokens in 2D windows (like Swin) is essential to preserve spatial structure. EVT proves that if your attention mechanism already possesses a strong 2D spatial prior (via the Euclidean decay matrix), you can safely flatten the image and group tokens in 1D.
  • Expanded Modeling Range: Mathematically, 1D grouping expands the effective receptive field coverage rate by a factor of $g$ (the number of groups) compared to 2D grouping. This allows tokens to interact across a longer range in fewer layers, while offering much better flexibility and adaptability when changing input image resolutions.
Fig. 4: Comparison between 2D grouping/shuffling and our 1D grouping/shuffling.

Key Results

EVT achieves an exceptional accuracy-speed trade-off across all scales:

  • ImageNet-1K Classification: EVT-T (15M parameters) achieves 83.0% top-1 accuracy, outperforming RMT-T by 0.6%. The large variant, EVT-L, reaches 86.6% top-1 accuracy at $384\times384$ resolution without requiring any extra training data.
  • Throughput & Efficiency: Thanks to the highly parallelizable 1D grouping strategy, EVT-T achieves a throughput of 2142 images/second on a single NVIDIA A100 GPU, significantly faster than RMT-T (1650 imgs/s) and Swin-T (1704 imgs/s).
Fig. 1: FLOPs v.s. Top-1 accuracy on ImageNet-1K.
  • Downstream Tasks: On MS-COCO object detection (using Mask R-CNN), EVT-B beats RMT-B by 1.1 $AP^b$. On ADE20K semantic segmentation (using UperNet), EVT-B achieves a 52.7% mIoU, outperforming many backbones that are an order of magnitude larger.

Limitations and Future Work

The primary limitation noted in the paper is that the decay coefficient $\gamma$ within the Euclidean matrix must be manually configured as a hyperparameter for each attention head to capture multi-scale features. Future work could explore adaptively learning these decay coefficients during training to make the spatial prior fully dynamic and data-dependent.

Summary updated

Abstract

In recent years, the Vision Transformer (ViT) has garnered significant attention within the computer vision community. However, the core component of ViT, Self-Attention, lacks explicit spatial priors and suffers from quadratic computational complexity, limiting its applicability. To address these issues, we have proposed RMT, a robust vision backbone with explicit spatial priors for general purposes. RMT utilizes Manhattan distance decay to introduce spatial information and employs a horizontal and vertical decomposition attention method to model global information. Building on the strengths of RMT, Euclidean enhanced Vision Transformer (EVT) is an expanded version that incorporates several key improvements. Firstly, EVT uses a more reasonable Euclidean distance decay to enhance the modeling of spatial information, allowing for a more accurate representation of spatial relationships compared to the Manhattan distance used in RMT. Secondly, EVT abandons the decomposed attention mechanism featured in RMT and instead adopts a simpler spatially-independent grouping approach, providing the model with greater flexibility in controlling the number of tokens within each group. By addressing these modifications, EVT offers a more sophisticated and adaptable approach to incorporating spatial priors into the Self-Attention mechanism, thus overcoming some of the limitations associated with RMT and further enhancing its applicability in various computer vision tasks. Extensive experiments on Image Classification, Object Detection, Instance Segmentation, and Semantic Segmentation demonstrate that EVT exhibits exceptional performance. Without additional training data, EVT achieves 86.6% top1-acc on ImageNet-1k.
BibTeX
@misc{fan2026advancing,
  author = {Qihang Fan and Huaibo Huang and Mingrui Chen and Hongmin Liu and Ran He},
  title = {{Advancing Vision Transformer with Enhanced Spatial Priors}},
  year = {2026},
  eprint = {2604.18549},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2604.18549},
}

Loading related content...

Figure