Abstract
Vision Transformers (ViTs) have revolutionized computer vision, yet their self-attention mechanism lacks explicit spatial inductive biases, leading to suboptimal performance on spatially-structured tasks. Existing approaches introduce data-independent spatial decay based on fixed distance metrics, applying uniform attention weighting regardless of image content and limiting adaptability to diverse visual scenarios. Inspired by recent advances in large language models where content-aware gating mechanisms significantly outperform static alternatives, we present the first successful adaptation of data-dependent spatial decay to 2D vision transformers. We introduce Spatial Decay Transformer (SDT), featuring a novel Context-Aware Gating (CAG) mechanism that generates dynamic, data-dependent decay for patch interactions. Our approach learns to modulate spatial attention based on both content relevance and spatial proximity. We address the fundamental challenge of 1D-to-2D adaptation through a unified spatial-content fusion framework that integrates manhattan distance-based spatial priors with learned content representations. Extensive experiments on ImageNet-1K classification and generation tasks demonstrate consistent improvements over strong baselines. Our work establishes data-dependent spatial decay as a new paradigm for enhancing spatial attention in vision transformers.
Introduction
Standard Vision Transformers (ViTs) suffer from a fundamental "spatial blindness." Because the self-attention mechanism treats an image as a sequence of flattened patches, it is inherently permutation equivariant—meaning it treats two adjacent pixels exactly the same as two pixels on opposite corners of the image. While humans intuitively understand that nearby pixels are typically more related than distant ones, ViTs must learn these relationships entirely from scratch, which is data-intensive and inefficient.
Previous attempts to fix this, like Retentive Networks (RMT), introduced static spatial decay. These models use a fixed mathematical rule (like Manhattan distance) to reduce the attention weight of distant patches. However, these rules are content-agnostic; they apply the same decay to a uniform blue sky as they do to a complex, multi-part object. This rigidity prevents the model from focusing on semantically relevant regions that might be far apart but highly related.
Methodology: Context-Aware Gating
The Spatial Decay Transformer (SDT) moves beyond static geometric rules toward a dynamic, content-aware understanding of space through two core innovations:
1. Context-Aware Gating (CAG)
Instead of a fixed decay rate, the model learns to generate "decay logits" based on the input features of the patches themselves. It uses a unified spatial-content fusion framework that combines Manhattan distance with these learned gates. As shown in the architecture diagram below, the CAG mechanism (Compute Decay) modulates the attention scores dynamically.
2. Symmetry and Efficiency
In 1D language modeling, decay is usually causal (looking backward). In 2D images, relationships are bidirectional. The authors found that averaging the gating vectors of patch $i$ and patch $j$ was essential for a balanced, reciprocal measure of mutual influence. To handle high-resolution images in early network stages where computing a full $L \times L$ mask is $O(L^2)$, the authors developed a "decomposed" version that separately computes horizontal and vertical 1D decays, reducing complexity to $O(H^2 + W^2)$.
Experimental Results
The model was evaluated on two primary benchmarks: ImageNet-1K for classification and 256x256 ImageNet for class-conditional image generation using a Diffusion Transformer (DiT) framework.
ImageNet-1K Classification
As detailed in the table below, the SDT-H-B (Base) model achieved 85.1% Top-1 accuracy on ImageNet-1K, outperforming the previous state-of-the-art spatial-bias model, RMT-B (85.0%), and standard hierarchical models like Swin-B (84.1%). The performance gains were consistent across all scales, from Tiny (14M parameters) to Base (54M parameters).
Image Generation
When used as the backbone for image generation, the SDT-P-XL-G achieved a Fréchet Inception Distance (FID) of 2.25, marginally outperforming the highly optimized DiT-XL-G (2.27) while maintaining a similar parameter count. This demonstrates that the CAG mechanism is not only effective for discriminative tasks but also for capturing the complex spatial distributions required for high-quality synthesis.
Discussion and Conclusion
The most significant insight from this work is that spatial proximity should not be a hard, static rule. By making decay data-dependent, the model can "turn off" the decay for distant patches if they are semantically related (e.g., the front and back wheels of a car), while suppressing irrelevant nearby noise. Furthermore, because the model starts with a sensible spatial prior that it can then refine, it trains significantly faster and reaches higher peak accuracy than models using static decay or no decay at all.
While the decomposed implementation helps with memory in early stages, the full 2D mask used in later stages still scales quadratically. Future work may explore extending this to 3D spatial-temporal decay for video understanding or optimizing the 2D mask for even higher resolutions. Ultimately, SDT establishes data-dependent spatial decay as a new paradigm for enhancing spatial attention in vision transformers.
Summary updated