Standard vision models, from Convolutional Neural Networks (CNNs) to Vision Transformers (ViTs), typically operate under a "uniformity assumption." They allocate the same amount of computation to every pixel or image patch, regardless of whether that region contains a critical object feature or an empty sky. As Boxi Wu et al. and Arash Vahdat et al. observe, this lack of prioritization leads to massive computational waste, as background information often consumes significant resources without aiding the model's final prediction.
The research direction of spatial redundancy elimination seeks to break this uniformity. By identifying and bypassing uninformative regions, these methods allow models to focus their "attention budget" where it matters most.
The Cost of Uniformity
In natural images, information is not distributed evenly. In tasks like human pose estimation, the relevant features often occupy only a small fraction of the total image area, yet standard transformers treat all patches with equal importance (Haoyu Ma et al.). Similarly, in 3D lane detection, Chengtang Yao et al. point out that mapping every pixel to a dense grid creates overhead because lanes occupy a tiny fraction of the scene; most of the map represents redundant background like the sky or empty road.
This redundancy is so prevalent that Christos Matsoukas et al. found that ViTs can maintain accuracy even when a significant majority of input patches are randomly discarded during training. This suggests a "Data Level Lottery Ticket Hypothesis": inside every image, there is a sparse subset of "winning" patches that are sufficient for the model to reach its target performance (Xuan Shen et al.).
Mechanism 1: Pruning and Dropping
The most direct way to exploit redundancy is to stop processing unimportant regions entirely. This is often achieved through token pruning.
- Static and Dynamic Pruning: Haoyu Ma et al. progressively prune background tokens that don't contain the subject. Arash Vahdat et al. introduce A-ViT, which adaptively halts computation for simple background patches during inference.
- Top-Down Selection: Instead of deciding what to drop at the start, Chao Xu et al. use a "Patch Slimming" approach, identifying what the final layer actually needs and pruning backward through the network.
- Significance Mapping: Sucheng Ren proposes SG-Former, which uses a significance map to keep more tokens for salient regions and fewer for the background, maintaining a global receptive field while cutting costs.
Mechanism 2: Token Merging and Clustering
Rather than deleting data, another strategy is to compress it. Because neighboring tokens in intermediate layers often represent nearly identical semantic features, they can be merged without losing information.
Weicong Liang et al. and Jin Li et al. (AiluRus) use spatial-aware clustering to merge redundant tokens in flat areas while preserving high-resolution detail at object boundaries. This is particularly effective for dense prediction tasks like segmentation, where the interior of a large object (like a building) is highly redundant, but its edges are critical for localization. In structured data like charts, Liang Zhang et al. show that merging redundant visual tokens allows compact models to rival much larger ones in understanding.
Mechanism 3: Adaptive Resolution and Foveation
A third approach mimics human biological vision: foveation. Humans see the center of their gaze in high resolution and the periphery in low resolution.
Matteo Tiezzi et al. introduce Foveated Convolutional Layers that prioritize computation at a "focus of attention." Similarly, the Quadformer by Tomer Ronen et al. replaces the uniform grid of ViTs with a "patch mosaic" of varying resolutions, guided by a saliency scorer. Gao Huang et al. formalize this as a "Glance and Focus" process, where a model first glances at a low-resolution image and only focuses on high-resolution patches when the initial pass is ambiguous.
Why It Works: The Sparsity of Information
The success of these methods stems from the inherent mathematical properties of visual data. Michaël Ramamonjisoa et al. demonstrate that depth maps are "piecewise flat," meaning they consist of large, smooth regions where dense pixel-wise processing is redundant. By estimating sparse wavelet coefficients instead of raw pixels, they slash computation by over 50%.
Furthermore, Yifan Jiang et al. show that natural images exhibit high local redundancy, allowing spatially-varying kernels to be shared across local patches. This suggests that the "intelligence" of a vision model doesn't come from processing every pixel, but from its ability to adaptively allocate its capacity to the high-entropy, informative regions of the scene.
Key Foundational Work
Selective Attention for Handwritten Digit Recognition (1995) — contributed 1 concept elements to this direction.