Vision Transformers (ViTs) have fundamentally changed computer vision by replacing the local sliding windows of Convolutional Neural Networks (CNNs) with global self-attention. This shift allows models to capture long-range dependencies—the "global context"—across an entire image simultaneously. However, this strength comes with a significant trade-off: a diminished ability to resolve fine-grained local details. As Xianghao Zang et al. observe in their work on pedestrian retrieval, standard ViTs often prioritize the "forest" at the expense of the "trees," losing the subtle textures and part-specific details necessary to distinguish between similar objects.
The Locality Gap
The core of the problem lies in how Transformers process visual data. Most ViTs divide an image into discrete, non-overlapping patches. While this makes the computational complexity manageable, Cheng Yuan 0002 argues in the context of surgical scene segmentation that this patch-based approach can obscure nuances within the patches themselves. Because the self-attention mechanism treats every patch as a token and calculates relationships across the whole image, it effectively acts as a low-pass filter.
This global focus creates a "performance ceiling" for tasks requiring geometric precision. For example, Shuo Wang et al. note in TSP-Transformer that there is a fundamental tension between high-level semantic understanding and the ultra-low-level spatial precision needed for edge localization. In medical imaging, this manifests as blurry or imprecise anatomical boundaries, a challenge highlighted by Jiongtong Hu in the EchoONE framework for echocardiography.
Mechanisms for Local Refinement
To bridge this gap, researchers are re-introducing "locality inductive bias"—the assumption that neighboring pixels are highly correlated—back into the Transformer architecture. These solutions generally fall into three categories:
1. Hybrid Parallelism Many researchers argue that the best way to handle the global-local trade-off is to let CNNs and Transformers work in tandem. Eileen Lee Ming Su et al. integrated a local CNN module into the Segment Anything Model (SAM) to capture high-resolution details that the global encoder overlooks. Similarly, Yanfeng Zhou et al. proposed nnWNet, which allows local and global information to flow through the network in parallel branches, preventing the "feature confusion" that occurs when these signals are prematurely fused.
2. Multi-Scale and Focal Attention Instead of attending to the whole image at once, some models restrict the attention field to smaller, overlapping regions. Chengyin Li et al. introduced FocalUNETR, which uses focal self-attention to prioritize local context for boundary-aware segmentation. In a different approach, Youngwan Lee et al. developed MPViT, a multi-path architecture that processes features at different scales simultaneously to preserve spatial correlations essential for dense prediction tasks like object detection.
3. Local Adapters and Prompts For large, pre-trained models where full retraining is impossible, researchers use lightweight "adapters." Ting Liu et al. demonstrated in MaPPER that adding specialized local convolution adapters to a frozen CLIP backbone significantly improves spatial precision in referring expression comprehension.
Why Locality Matters in Practice
The necessity of these local mechanisms is most visible in high-stakes domains. In deepfake detection, Dat Nguyen et al. found that global attention tends to smooth over the subtle blending artifacts that characterize forgeries; their FakeFormer model uses local attention to force the model to focus on these "vulnerable patches."
In robotic surgery, Pengfei Hao et al. noted that the loss of fine details makes it difficult for AI to establish dependencies between a surgeon's textual query and the state of small surgical tools. By restoring local awareness, these models move beyond broad scene understanding toward the high-precision localization required for real-world interaction. As Khaled Alomar et al. summarize in their survey of action recognition, the future of vision backbones likely lies in these hybrid models that combine the long-range reasoning of Transformers with the granular efficiency of local feature extraction.
Key Foundational Work
SwinTransFuse: Fusing swin and multiscale transformers for fine-grained image recognition and retrieval (2022) — contributed 2 concept elements to this direction.
Dual-stream Network for Visual Recognition (2021) — contributed 1 concept elements to this direction.
CSU-Net: A CNN-Transformer Parallel Network for Multimodal Brain Tumour Segmentation (2022) — contributed 1 concept elements to this direction.