Overlapping Patch Embeddings for Vision Transformers

48 concepts39 papers1 proposal

The standard Vision Transformer (ViT) architecture begins by slicing an image into a grid of non-overlapping patches—typically $16 \times 16$ pixels—and projecting each patch linearly into a token vector. While computationally clean, this rigid partitioning introduces an immediate structural flaw: it breaks spatial continuity. If a critical visual feature, such as a sharp edge or a fine geometric curve, happens to fall across the boundary of two patches, its information is abruptly bisected.

As shown by Cheng et al. in Gaze Estimation using Transformer, this hard-splitting mechanism degrades performance in high-precision regression tasks because it disrupts fine-grained spatial relationships and local geometric structures right at the input layer. To fix this, a powerful architectural modification has emerged: overlapping patch embedding.

The Mechanics of Overlapping Tokenization

The core intuition behind overlapping patch embedding is simple: allow adjacent patches to share a "halo" of pixel information. Instead of forcing the tokenization layer's stride to equal its kernel size, models configure the stride to be smaller than the kernel size.

This is typically achieved by replacing the standard linear projection layer with a convolutional layer or an unfold operation. For instance, Shao et al. demonstrated in PVT v2: Improved baselines with Pyramid Vision Transformer that using a convolutional layer with a stride smaller than its kernel size creates overlapping tokens that inherently preserve the visual relationships between adjacent pixels at patch boundaries. Similarly, Wu et al. in CvT: Introducing Convolutions to Vision Transformers leveraged overlapping convolutional layers to capture local spatial dependencies at the earliest stages of the network, facilitating a smoother, hierarchical reduction in token count.

Mathematically, if a standard ViT uses a kernel size of $P \times P$ and a stride of $P$, an overlapping patch embedding might use a kernel size of $K \times K$ and a stride of $S$, where $S < K$. The resulting token sequence contains redundant, shared pixel data along the borders, ensuring that no feature is completely isolated by an arbitrary grid line.

Preserving Spatial Continuity and Mitigating Artifacts

Why does this redundant pixel sharing matter so much? The primary benefit is the preservation of spatial continuity. When feature maps undergo downsampling or tokenization, maintaining a continuous landscape of information prevents the network from developing blind spots at patch borders.

In SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers, Xie et al. observed that using overlapping patches during downsampling preserves local spatial continuity that is otherwise lost with non-overlapping patch merging. This ensures that the boundaries between patches remain well-represented, yielding highly coherent feature maps across different stages of the network.

This continuity is especially vital in dense prediction and restoration tasks where boundary discontinuities manifest as visible defects. For example, Perera et al. showed in Transformer-based SAR Image Despeckling that processing overlapping regions avoids the grid-like artifacts typically caused by independent patch processing. In the domain of image restoration, Chu et al. developed a method in Improving Image Restoration by Revisiting Global Information Aggregation that processes feature maps using overlapping local windows, allowing internal network layers to interact smoothly across window boundaries and eliminating distribution shifts during inference.

Interestingly, the intuition that overlapping windows mitigate boundary noise is not entirely new. Even in early handwriting recognition systems, such as the work by Parizeau et al. in Character Recognition Experiments Using Unipen Data (ICDAR 2001), feature extraction utilized overlapping spatial regions specifically to reduce representation discontinuities when a stroke segment shifted slightly between adjacent cells.

Capturing Fine-Grained and Morphological Cues

For tasks that rely on subtle, highly localized visual cues—such as fine-grained classification, face recognition, or object tracking—rigid patch boundaries can fragment the very features the model needs to detect.

Consider a fine-grained classification task where the distinguishing feature is a tiny pattern on a bird's wing. As argued by He et al. in TransFG: A Transformer Architecture for Fine-Grained Recognition, utilizing a sliding window to generate overlapping image patches prevents the fragmentation of small, discriminative visual features that would otherwise be bisected by standard non-overlapping grids.

This principle applies directly to structured images like faces or biological structures:

By ensuring that small features remain intact within at least one token's receptive field, overlapping embeddings grant the model translation and rotation invariance, a property explicitly highlighted by Yang et al. in CrowdFormer: An Overlap Patching Vision Transformer for Top-Down Crowd Counting.

Inductive Biases and Data Efficiency

Pure Vision Transformers are notoriously "data-hungry" because they lack the structural inductive biases inherent to convolutional networks, such as local translation invariance. Overlapping patch embeddings act as a bridge, injecting a soft spatial prior directly into the transformer architecture.

In a data-driven architectural search, Chen et al. discovered in Auto-scaling Vision Transformers without Training that vision transformers demonstrate an inherent structural preference for overlapping tokenization (where kernel size exceeds stride) to better capture spatial coherency.

This structural choice drastically improves data efficiency. Liu et al. showed in Bridging the Gap Between Vision Transformers and Convolutional Neural Networks on Small Datasets that generating overlapping patches helps bridge the performance gap between convolutional networks and transformers on small-scale benchmarks. By maintaining high-resolution feature relationships from the earliest layers, the model does not have to spend its parameter capacity learning how to stitch fragmented patch boundaries back together. Similarly, Chen et al. in EffLoc: Lightweight Vision Transformer for Efficient 6-DOF Camera Relocalization leveraged this spatial sensitivity to capture local dependencies and fine-grained details, enabling lightweight models to achieve high-precision camera relocalization.

Ultimately, overlapping patch embedding shifts the tokenization process from a harsh geometric dissection to a smooth, continuous sampling operation, allowing subsequent self-attention layers to operate on a richer, more structurally sound representation of the visual world.

Key Foundational Work

Visual Objects Classification with Sliding Spatial Pyramid Matching (2013) — contributed 1 concept elements to this direction.
Character Recognition Experiments Using Unipen Data (2001) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_overlapping_patch_embedding_542bbcbf,
  title = {overlapping patch embedding},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/overlapping-patch-embedding/rdr_542bbcbf0bab44ae842d191bc25580af},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_overlapping_patch_embedding_542bbcbf_v1,
  title = {overlapping patch embedding},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/overlapping-patch-embedding/rdr_542bbcbf0bab44ae842d191bc25580af/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake