Standard Vision Transformers (ViTs) treat images as a rigid grid of flattened patches. To retain spatial awareness, they add static, absolute positional encodings—predefined coordinate vectors—to each patch token. While this works well for fixed-size inputs, it creates a fundamental architectural bottleneck: if the input resolution or aspect ratio changes at inference time, the coordinate system breaks.
When a model trained on a $14 \times 14$ grid encounters a $28 \times 28$ grid, the sequence length quadruples, forcing researchers to rely on manual adjustments like bilinear or bicubic interpolation to stretch the positional embedding grid (Park et al., Lin et al.). However, interpolation alters the frequency and scale of the learned spatial signals, often leading to catastrophic performance degradation.
To overcome this "grid lock," a prominent research direction focuses on resolution-agnostic spatial encodings. Instead of hardcoding static coordinates, these techniques allow vision models to dynamically compute or extrapolate spatial relationships, enabling seamless zero-shot generalization to arbitrary image sizes and aspect ratios.
Implicit Spatial Encoding via Convolutions
One of the most elegant solutions replaces explicit coordinate vectors with local operations. Chu et al. pioneered this approach by introducing Conditional Positional Encodings (CPE) generated via a Position Encoding Generator (PEG). Instead of adding a fixed-length vector to the tokens, a PEG applies a local operator—such as a depth-wise convolution—directly to the input feature map.
Standard ViT: [Patch Tokens] + [Fixed Coordinate Grid] = Spatial Tokens
CPE / PEG: [Patch Tokens] ──> [Depth-wise Conv] ──> Spatial Tokens
This mechanism leverages the inherent inductive biases of convolutions: locality and translation equivariance. A $3 \times 3$ convolutional kernel operates on a token's immediate neighborhood regardless of whether the total image is $224 \times 224$ or $1024 \times 1024$. As a result, the generated spatial representation remains stable across varying sequence lengths and image dimensions without requiring any interpolation.
The Power of the Zero-Padding Anchor
A natural question arises: if convolutions are translation-equivariant, how does the model learn absolute positions (e.g., distinguishing the top-left corner from the center)?
The secret lies in the boundary conditions. As demonstrated by Xu et al., the standard zero-padding used in convolutional layers unintentionally acts as a spatial anchor. Tokens near the edges of an image receive a unique signal because their receptive fields overlap with the padded zeros. This boundary information propagates inward across subsequent layers, allowing the network to infer both relative and absolute positions implicitly.
Building on this intuition, the authors of SANA showed that explicit positional embeddings can be omitted entirely in transformer architectures by utilizing zero-padded convolutional layers within the feed-forward blocks. This implicit spatial awareness provides several key advantages:
- Data Efficiency: It introduces crucial structural biases like locality, which helps vision models capture local structures more effectively when training data is limited (Zhang).
- Operational Flexibility: It allows transformer-based encoders to process images of varying resolutions without coordinate interpolation, maintaining structural consistency across diverse image sizes (Perera et al., Li et al.).
Continuous and Rotary 2D Extrapolation
An alternative strategy abandons discrete grids in favor of continuous mathematical manifolds. Rather than treating positions as discrete indices ($1, 2, 3...$), these methods define space as a smooth, continuous field, allowing the model to evaluate coordinates it has never seen during training.
2D Rotary Position Embeddings (RoPE)
Originally designed for Large Language Models to handle long textual contexts, Rotary Position Embeddings (RoPE) apply a rotation matrix to the query and key vectors in the self-attention mechanism. The core mathematical beauty of RoPE is that the inner product of two rotated vectors depends strictly on their relative distance.
To adapt this to computer vision, researchers have extended RoPE to two dimensions. Heo et al. introduced a "RoPE-Mixed" approach that utilizes learnable frequencies to capture 2D spatial relationships. By treating space as a continuous rotation, the attention mechanism naturally generalizes to higher resolutions during inference—a property known as resolution extrapolation.
Discrete Grid: [Pos 1] [Pos 2] [Pos 3] ──> Cannot extrapolate to [Pos 100]
Continuous Manifold: θ_x, θ_y Rotation Angles ──> Smoothly evaluates any coordinate
Similarly, Chu et al. developed AS2DRoPE (Auto-Scaled 2D RoPE) for VisionLLaMA, incorporating auto-scaled interpolation to preserve spatial coherence when scaling up resolutions. Because the underlying geometry remains consistent, models can leverage these rotary biases to generalize to much higher upsampling ratios at inference time than those encountered during training (Couairon et al.).
Continuous Relative Biases and Proportions
Other continuous methods modify the attention matrix directly or redefine coordinates as fractions:
- Log-Spaced Continuous Bias: Chen et al. mapped relative coordinates into a log-space to construct a continuous relative position bias. This logarithmic compression bounds the coordinate space, improving the transferability of learned spatial relationships across vastly different window sizes and scales.
- Proportion-Based Embeddings: To handle non-square aspect ratios without distortion, Gallo Fernández et al. designed embeddings that capture the relative position of patches using proportions of the total image dimensions rather than absolute patch indices. Because a patch at the horizontal midpoint always has a coordinate of $0.5$ (regardless of whether the row contains 14 or 40 patches), the encoding becomes invariant to the total patch count.
Decoupling Space from Attention
A final architectural perspective argues that positional encodings should be decoupled from the core attention calculation entirely. In standard ViTs, positional embeddings are added directly to the token features, meaning spatial coordinates and semantic content are mixed together before computing queries, keys, and values.
Yu bypassed this mixing in the CSWin Transformer by applying a depth-wise convolution branch directly to the value component of the attention mechanism.
$$\text{Attention}(Q, K, V) = \text{Softmax}\left(\frac{QK^T}{\sqrt{d}}\right)V + \text{DepthWiseConv}(V)$$
By injecting spatial information directly into the values rather than modifying the attention scores, the model separates "what" the feature is from "where" it is located. This structural decoupling prevents the attention matrix from becoming distorted when image dimensions vary significantly from the training set, allowing the architecture to adapt more effectively to arbitrary input resolutions in downstream tasks.
Taking decoupling a step further, Xiao et al. introduced adaptive spatial tokens that completely separate spatial understanding from feature extraction. By unlinking spatial tokens from the fixed image grid, the architecture experiences minimal performance degradation during zero-shot scaling, offering a robust foundation for truly resolution-agnostic vision models.