Transformers have a well-known "computational tax": the self-attention mechanism scales quadratically with the number of tokens. To process high-resolution images, long videos, or extensive text prompts efficiently, reducing the token count is a necessity.
However, a naive approach to token pruning introduces a hidden failure mode. When uninformative tokens are discarded, the remaining tokens are typically packed into a continuous sequence and re-indexed (e.g., [0, 1, 2...]). This seemingly harmless compression step tears the underlying coordinate system of the input. Because Transformers rely entirely on positional embeddings to understand sequence order and spatial geometry, re-indexing scrambles the relative distances the model learned during training.
To solve this, a new paradigm has emerged: Positional Index Preservation. The core intuition is simple: when compressing or pruning a sequence, the surviving tokens must retain their original coordinate identifiers or structural layout. By preserving this spatial and sequential fabric, models can operate on highly sparse inputs without losing their capacity for complex reasoning, visual grounding, or physical interaction.
The Pitfall of Re-Indexing
To understand why preserving original indices works, we must first look at how Transformers process distance. In language models, the relative distance between tokens dictates syntactic and semantic relationships.
As shown by Muhammad Adnan et al. in Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inference, re-indexing a pruned Key-Value (KV) cache to appear continuous disrupts the model's internal representation, leading to severe accuracy degradation. Conversely, retaining the original positional indices of tokens after pruning preserves the model's accuracy while slashing memory footprints.
This phenomenon holds true during the initial prompt processing phase as well. Jingyu Liu et al. demonstrate in Speculative Prefill: Turbocharging TTFT with Lightweight and Training-Free Token Importance Estimation that keeping the original position IDs of selected tokens allows a model to understand the relative structure of a prompt even when up to 90% of the tokens are removed.
When these positional components are stripped or corrupted, the model's capacity for sequence understanding collapses entirely. In their analysis of highly sparse models, Eldar Kurtic et al. (How to Prune Your Language Model: Recovering Accuracy on the ``Sparsity May Cry'' Benchmark) note that maintaining dense positional embeddings during pruning is what prevents a model from becoming position-agnostic, preserving its core reasoning capabilities.
Preserving the 2D Spatial Manifold
The challenge becomes even more acute in Vision Transformers (ViTs) and Multimodal Large Language Models (MLLMs), where tokens represent patches of a 2D image or video frame. Standard token pruning techniques that remove uninformative patches entirely disrupt the 2D spatial grid. As Yifan Xu et al. observe in Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformer, this structural disruption prevents pruned tokens from being compatible with hierarchical or deep-narrow architectures that rely on consistent spatial layouts. To bypass this, they propose using placeholder tokens that receive coarse, low-cost updates rather than deleting background patches outright, maintaining the integrity of the 2D manifold.
When explicit token removal is required, maintaining a persistent coordinate system is paramount for downstream tasks that require precise spatial awareness:
- Graphical User Interfaces (GUIs): In ShowUI: One Vision-Language-Action Model for GUI Visual Agent, Kevin Qinghong Lin et al. show that selecting a subset of original visual tokens—rather than merging or pooling them—preserves the exact integrity of positional embeddings. This allows a visual agent to ground coordinates precisely and click on tiny interactive UI elements.
- Robotic Manipulation: Robotic models are highly sensitive to the physical layout of their environment. Ye Li et al. argue in SP-VLA: A Joint Model Scheduling and Token Pruning Approach for VLA Model Acceleration that standard semantic pruning risks breaking the spatial grid. They utilize an order-preserving union of selected tokens to ensure the underlying physical relationships remain intact.
- Dynamic Vision Pruning: To maintain spatial context across deep networks, Hao Wu et al. introduce persistent positional encoding in HiDrop: Hierarchical Vision Token Reduction in MLLMs via Late Injection, Concave Pyramid Pruning, and Early Exit. This mechanism ensures that surviving tokens retain their original coordinate-based identifiers throughout dynamic reduction stages, preventing geometric context loss.
Topology Preservation vs. Semantic Selection
A key insight across recent research is that preserving the global layout—the "visual skeleton"—is often more critical for spatial reasoning than keeping the most semantically rich individual tokens.
Yihong Huang et al. demonstrate this clearly in Nüwa: Mending the Spatial Integrity Torn by VLM Token Pruning. They show that partitioning visual tokens into a uniform grid before pruning frequently outperforms sophisticated semantic selection strategies. Uniform spatial reduction ensures that the global layout is preserved across the image, preventing the collapse of visual grounding performance.
Naive Pruning: [Patch A] [Patch B] [Patch C] --> Packed & Re-indexed: [0, 1, 2] (Spatial Layout Lost)
Topology Pruning: [Patch A] [ Empty ] [Patch C] --> Retained Coordinates: [(0,0), (0,2)] (Spatial Layout Preserved)
This concept of topology preservation is leveraged across several domains:
- Multimodal Abstraction: Linli Yao et al. (DeCo: Decoupling Token Compression from Semantic Abstraction in Multimodal Large Language Models) show that simple, grid-preserving spatial pooling prevents the scrambling of relative positional information between image patches, vastly improving object localization compared to learned query-based compression.
- Document Understanding: Parsing tables, charts, and forms requires strict adherence to layout structure. Anwen Hu et al. design the H-Reducer module in mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understanding specifically to compress token counts while preserving the fine-grained spatial relationships essential for structural learning.
- Video Speculative Decoding: When accelerating video models, the authors of SpecVLM: Enhancing Speculative Decoding of Video LLMs via Verifier-Guided Token Pruning find that uniform spatial reduction of low-importance tokens preserves the visual skeleton necessary for coherent sequence generation, proving more vital than preserving fine-grained temporal details.
- Extreme Video Compression: Even when compressing video sequences down to a fraction of a percent of their original length, Haichao Zhang et al. (VQToken: Neural Discrete Token Representation Learning for Extreme Token Reduction in Video Large Language Models) utilize an indexing mechanism to maintain original spatial and temporal coordinates, keeping complex downstream reasoning possible.
Mechanisms for Structural Preservation
To implement index and structure preservation without sacrificing the speed gains of token reduction, researchers have developed several distinct architectural mechanisms:
1. Spatial Window Initialization
To make token reduction compatible with local-attention architectures that rely on fixed spatial grids, Ming Lin et al. (Making Vision Transformers Efficient from A Token Sparsification View) incorporate spatial window initialization into the token reduction process. This anchors the sparsification process within a structured geometric framework.
2. Dynamic Embedding Remapping
In generative contexts like diffusion language models, dropping tokens can cause a loss of spatial coherence. Xinhua Chen et al. demonstrate in DPad: Efficient Diffusion Language Models with Suffix Dropout that rotary positional embeddings (RoPE) can be dynamically remapped to reflect the true relative distances between the remaining elements after suffix tokens are dropped.
3. Spatial Rearrangement (Pixel Unshuffling)
For long-horizon embodied navigation, models must retain environmental landmarks over hundreds of frames. Instead of dropping tokens, Botao Ren et al. (AstraNav-Memory: Contexts Compression for Long Memory) utilize pixel unshuffling to shift local spatial information directly into the channel dimension. This compression technique condenses visual tokens via convolutional layers while preserving the underlying topological structure of the scene.
4. Language-Guided Coordinate Retention
Even when external modalities dictate token importance, positional integrity must be maintained. Yizheng Sun et al. show in LVPruning: An Effective yet Simple Language-Guided Vision Token Pruning Approach for Multi-modal Large Language Models that using language prompts to prune up to 90% of irrelevant visual tokens succeeds only if the surviving patches retain their original spatial coordinates, allowing the model to correctly interpret geometric relationships.
By shifting the focus from what tokens to remove to how the remaining tokens remember their origins, positional index preservation allows modern Transformers to achieve extreme efficiency gains without sacrificing structural intelligence.
Key Foundational Work
Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformer (2022) — contributed 3 concept elements to this direction.
Exploiting Spatial Sparsity for Event Cameras with Visual Transformers (2022) — contributed 1 concept elements to this direction.
Making Vision Transformers Efficient from A Token Sparsification View (2023) — contributed 1 concept elements to this direction.