For decades, computer vision was dominated by the sliding window. Convolutional Neural Networks (CNNs) processed images through local receptive fields, gradually building a global understanding by stacking layers. However, a fundamental shift has occurred: treating images not as a grid of pixels to be filtered, but as a sequence of patches to be attended to. This patch-based self-attention mechanism allows models to capture long-range dependencies and global context from the very first layer.
Beyond the Sliding Window
The core limitation of traditional convolutional architectures is their inherent locality. A convolution operation only sees a small neighborhood of pixels at a time. To understand the relationship between a car in the foreground and a traffic light in the distance, a CNN must pass information through many successive layers.
In contrast, patch-based self-attention—the foundation of the Vision Transformer (ViT)—decomposes an image into a grid of non-overlapping patches. As Wu explains in a survey of multimodal models, these patches are projected into embeddings and treated as a sequence of tokens, much like words in a sentence. Because every token in a Transformer can attend to every other token, the model establishes a global receptive field immediately. Muhammad et al. note that this is particularly vital for autonomous driving, where interpreting complex, long-range spatial relationships is necessary for safe navigation.
The Intuition of Global Receptive Fields
Why does this global view matter? In tasks like medical imaging or document analysis, the meaning of a local feature often depends on distant context. For example, Yalcinbas et al. demonstrate that ViTs can effectively reconstruct MRI images by capturing long-range spatial dependencies that traditional methods struggle to resolve. Similarly, Souibgui et al. show that a "pure" transformer architecture, operating directly on pixel patches, excels at document enhancement by maintaining a global understanding of the page layout.
This mechanism also changes how models perceive objects. CNNs are notoriously "texture-biased," often identifying a "cat" based on fur patterns rather than the overall shape. Heinert et al. argue that Vision Transformers exhibit a higher degree of inherent shape awareness. By using self-attention to model structural relationships across the entire image, ViTs remain more stable when local textures are removed or distorted. This robustness is further evidenced in person re-identification; Xu and Zhu find that global self-attention captures invariant cues that allow models to recognize individuals even after they change their clothes.
Scaling and Dense Prediction
The flexibility of the patch-based approach allows it to scale across diverse tasks. In dense prediction tasks like depth estimation and semantic segmentation, Bochkovskiy et al. demonstrate that replacing convolutional backbones with ViTs leads to more detailed and globally coherent predictions. Because the Transformer maintains a constant feature resolution and global connectivity, it avoids the structural collapse that can occur in fixed-resolution convolutional networks.
Recent innovations have further refined how these patches are processed:
- Token Projection: Jiang et al. propose using multi-head self-attention within the projection head itself to better refine global representations in few-shot scenarios.
- Token Stacking: To handle high resolutions without exploding computational costs, Meng suggests partitioning visual information into subsets and injecting them into successive layers, a method called DeepStack.
- Data Augmentation: Zhao et al. found that ViTs benefit most from "MaskMix," a strategy that mixes images using a scattered, grid-like patch pattern, forcing the model to integrate information from distributed context.
A Unified Visual Language
Perhaps the most significant impact of patch-based attention is the unification of vision and language. By tokenizing images, researchers have made visual data compatible with the powerful Transformer architectures used in Large Language Models. As Huang et al. observe in their G-VUE benchmark, Transformer-based models pre-trained on vision-language data are currently the closest approximation to a "general-purpose" visual backbone. Whether the task is classifying ancient coins Reid et al., detecting misogynistic memes Mishra and Suryavardan, or editing images with diffusion models Feng et al., the underlying principle remains the same: global context is best captured when the model is free to look at the whole picture at once.