Abstract
Transformers are quickly becoming one of the most heavily applied deep learning architectures across modalities, domains, and tasks. In vision, on top of ongoing efforts into plain transformers, hierarchical transformers have also gained significant attention, thanks to their performance and easy integration into existing frameworks. These models typically employ localized attention mechanisms, such as the sliding-window Neighborhood Attention (NA) or Swin Transformer's Shifted Window Self Attention. While effective at reducing self attention's quadratic complexity, local attention weakens two of the most desirable properties of self attention: long range inter-dependency modeling, and global receptive field. In this paper, we introduce Dilated Neighborhood Attention (DiNA), a natural, flexible and efficient extension to NA that can capture more global context and expand receptive fields exponentially at no additional cost. NA's local attention and DiNA's sparse global attention complement each other, and therefore we introduce Dilated Neighborhood Attention Transformer (DiNAT), a new hierarchical vision transformer built upon both. DiNAT variants enjoy significant improvements over strong baselines such as NAT, Swin, and ConvNeXt. Our large model is faster and ahead of its Swin counterpart by 1.5% box AP in COCO object detection, 1.3% mask AP in COCO instance segmentation, and 1.1% mIoU in ADE20K semantic segmentation. Paired with new frameworks, our large variant is the new state of the art panoptic segmentation model on COCO (58.2 PQ) and ADE20K (48.5 PQ), and instance segmentation model on Cityscapes (44.5 AP) and ADE20K (35.4 AP) (no extra data). It also matches the state of the art specialized semantic segmentation models on ADE20K (58.2 mIoU), and ranks second on Cityscapes (84.5 mIoU) (no extra data).
Introduction
In the world of computer vision, Transformers have emerged as powerful alternatives to the long-reigning Convolutional Neural Networks (CNNs). However, the standard Transformer's self-attention mechanism scales quadratically with image resolution, creating a computational bottleneck. To address this, researchers introduced local attention mechanisms that limit a pixel's "vision" to a small nearby window. While these models are efficient, they often struggle to capture the global context necessary for complex visual tasks.
Dilated Neighborhood Attention (DiNA)
The core innovation of this work is Dilated Neighborhood Attention (DiNA). Building upon Neighborhood Attention (NA)—where each pixel attends to its immediate $k \times k$ neighbors—DiNA introduces a dilated pattern similar to dilated convolutions. By skipping pixels in its neighborhood, a DiNA layer can look at a grid spread across a much larger area without increasing the number of sampled points.
By alternating between NA layers (for local details) and DiNA layers (for global context), the DiNAT architecture achieves an exponential growth in receptive field size. Unlike methods like Swin's shifted windows, DiNA maintains a symmetric neighborhood around every pixel, preserving translational equivariance—a key inductive bias that contributes to the success of CNNs.
Implementation and Efficiency
To ensure practical efficiency, the authors extended the Neighborhood Attention Extension (NATTEN) with custom CUDA kernels. Theoretically, DiNA has the exact same number of floating-point operations (FLOPs) and parameters as standard NA. In practice, DiNAT variants are often faster than their Swin counterparts while maintaining a smaller memory footprint, despite the sparse memory access patterns required by dilation.
Key Results
DiNAT consistently outperforms strong baselines like Swin and ConvNeXt across a variety of challenging benchmarks:
- Panoptic Segmentation: DiNAT-L paired with Mask2Former achieved 58.2 PQ on COCO and 48.5 PQ on ADE20K, setting new state-of-the-art records at the time of publication.
- Object Detection: On COCO, the Large variant (DiNAT-L) outperformed its Swin counterpart by 1.5% box AP.
- Semantic Segmentation: On ADE20K, it reached 58.2 mIoU, matching or exceeding specialized state-of-the-art models.
Conclusion
DiNAT successfully bridges the gap between the efficiency of local attention and the global reach of full self-attention. By capturing long-range dependencies at no additional computational cost, it provides a flexible and powerful backbone for modern computer vision. The authors suggest that this local-plus-sparse-global pattern could be further extended to other domains, such as long-document processing in NLP or large-scale audio analysis.
Summary updated