Positional Index Preservation for Efficient Transformers

37 concepts23 papers1 proposal

Transformers have a well-known "computational tax": the self-attention mechanism scales quadratically with the number of tokens. To process high-resolution images, long videos, or extensive text prompts efficiently, reducing the token count is a necessity.

However, a naive approach to token pruning introduces a hidden failure mode. When uninformative tokens are discarded, the remaining tokens are typically packed into a continuous sequence and re-indexed (e.g., [0, 1, 2...]). This seemingly harmless compression step tears the underlying coordinate system of the input. Because Transformers rely entirely on positional embeddings to understand sequence order and spatial geometry, re-indexing scrambles the relative distances the model learned during training.

To solve this, a new paradigm has emerged: Positional Index Preservation. The core intuition is simple: when compressing or pruning a sequence, the surviving tokens must retain their original coordinate identifiers or structural layout. By preserving this spatial and sequential fabric, models can operate on highly sparse inputs without losing their capacity for complex reasoning, visual grounding, or physical interaction.


The Pitfall of Re-Indexing

To understand why preserving original indices works, we must first look at how Transformers process distance. In language models, the relative distance between tokens dictates syntactic and semantic relationships.

As shown by Muhammad Adnan et al. in Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inference, re-indexing a pruned Key-Value (KV) cache to appear continuous disrupts the model's internal representation, leading to severe accuracy degradation. Conversely, retaining the original positional indices of tokens after pruning preserves the model's accuracy while slashing memory footprints.

This phenomenon holds true during the initial prompt processing phase as well. Jingyu Liu et al. demonstrate in Speculative Prefill: Turbocharging TTFT with Lightweight and Training-Free Token Importance Estimation that keeping the original position IDs of selected tokens allows a model to understand the relative structure of a prompt even when up to 90% of the tokens are removed.

When these positional components are stripped or corrupted, the model's capacity for sequence understanding collapses entirely. In their analysis of highly sparse models, Eldar Kurtic et al. (How to Prune Your Language Model: Recovering Accuracy on the ``Sparsity May Cry'' Benchmark) note that maintaining dense positional embeddings during pruning is what prevents a model from becoming position-agnostic, preserving its core reasoning capabilities.


Preserving the 2D Spatial Manifold

The challenge becomes even more acute in Vision Transformers (ViTs) and Multimodal Large Language Models (MLLMs), where tokens represent patches of a 2D image or video frame. Standard token pruning techniques that remove uninformative patches entirely disrupt the 2D spatial grid. As Yifan Xu et al. observe in Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformer, this structural disruption prevents pruned tokens from being compatible with hierarchical or deep-narrow architectures that rely on consistent spatial layouts. To bypass this, they propose using placeholder tokens that receive coarse, low-cost updates rather than deleting background patches outright, maintaining the integrity of the 2D manifold.

When explicit token removal is required, maintaining a persistent coordinate system is paramount for downstream tasks that require precise spatial awareness:


Topology Preservation vs. Semantic Selection

A key insight across recent research is that preserving the global layout—the "visual skeleton"—is often more critical for spatial reasoning than keeping the most semantically rich individual tokens.

Yihong Huang et al. demonstrate this clearly in Nüwa: Mending the Spatial Integrity Torn by VLM Token Pruning. They show that partitioning visual tokens into a uniform grid before pruning frequently outperforms sophisticated semantic selection strategies. Uniform spatial reduction ensures that the global layout is preserved across the image, preventing the collapse of visual grounding performance.

Naive Pruning:      [Patch A] [Patch B] [Patch C]  --> Packed & Re-indexed: [0, 1, 2] (Spatial Layout Lost)
Topology Pruning:   [Patch A] [  Empty  ] [Patch C]  --> Retained Coordinates: [(0,0), (0,2)] (Spatial Layout Preserved)

This concept of topology preservation is leveraged across several domains:


Mechanisms for Structural Preservation

To implement index and structure preservation without sacrificing the speed gains of token reduction, researchers have developed several distinct architectural mechanisms:

1. Spatial Window Initialization

To make token reduction compatible with local-attention architectures that rely on fixed spatial grids, Ming Lin et al. (Making Vision Transformers Efficient from A Token Sparsification View) incorporate spatial window initialization into the token reduction process. This anchors the sparsification process within a structured geometric framework.

2. Dynamic Embedding Remapping

In generative contexts like diffusion language models, dropping tokens can cause a loss of spatial coherence. Xinhua Chen et al. demonstrate in DPad: Efficient Diffusion Language Models with Suffix Dropout that rotary positional embeddings (RoPE) can be dynamically remapped to reflect the true relative distances between the remaining elements after suffix tokens are dropped.

3. Spatial Rearrangement (Pixel Unshuffling)

For long-horizon embodied navigation, models must retain environmental landmarks over hundreds of frames. Instead of dropping tokens, Botao Ren et al. (AstraNav-Memory: Contexts Compression for Long Memory) utilize pixel unshuffling to shift local spatial information directly into the channel dimension. This compression technique condenses visual tokens via convolutional layers while preserving the underlying topological structure of the scene.

4. Language-Guided Coordinate Retention

Even when external modalities dictate token importance, positional integrity must be maintained. Yizheng Sun et al. show in LVPruning: An Effective yet Simple Language-Guided Vision Token Pruning Approach for Multi-modal Large Language Models that using language prompts to prune up to 90% of irrelevant visual tokens succeeds only if the surviving patches retain their original spatial coordinates, allowing the model to correctly interpret geometric relationships.

By shifting the focus from what tokens to remove to how the remaining tokens remember their origins, positional index preservation allows modern Transformers to achieve extreme efficiency gains without sacrificing structural intelligence.

Key Foundational Work

Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformer (2022) — contributed 3 concept elements to this direction.
Exploiting Spatial Sparsity for Event Cameras with Visual Transformers (2022) — contributed 1 concept elements to this direction.
Making Vision Transformers Efficient from A Token Sparsification View (2023) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_positional_index_preservation_08ae4eb0,
  title = {positional index preservation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/positional-index-preservation/rdr_08ae4eb0c1f24868b26f6090589b6cc8},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_positional_index_preservation_08ae4eb0_v1,
  title = {positional index preservation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/positional-index-preservation/rdr_08ae4eb0c1f24868b26f6090589b6cc8/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake