Preserving Spatial Structure in Visual Token Compression

19 concepts16 papers1 proposal

Multimodal Large Language Models (MLLMs) are increasingly tasked with processing high-resolution images and long video sequences. However, feeding thousands of visual tokens into a Large Language Model (LLM) creates a massive computational bottleneck due to the quadratic complexity of self-attention. To scale these models, a prominent research direction focuses on visual token compression—reducing the visual token budget before it reaches the LLM backbone.

While token compression drastically improves throughput and reduces latency, it introduces a fundamental tension: aggressive compression frequently destroys the spatial precision, geometric continuity, and fine-grained details required for complex reasoning. Understanding why this spatial degradation occurs—and how to prevent it—is a core challenge in modern multimodal AI.

Why Compression Destroys Spatial Logic

To understand why spatial reasoning degrades, we must look at how visual data is structured. Vision Transformers (ViTs) slice an image into a grid of patches, assigning each patch a discrete positional embedding. This grid forms a continuous coordinate landscape that allows the model to map relative distances, object boundaries, and geometric layouts.

When we compress these tokens using naive methods like uniform spatial pooling, we average vectors across a localized region. As Junyan Lin et al. demonstrate, while average pooling is highly effective for coarse perception and general semantic understanding, it strips away the spatial resolution required for precise localization. Blending distinct positional signatures makes it mathematically impossible for the downstream LLM to pinpoint exactly where a feature originated.

Furthermore, Linli Yao et al. argue that simple pooling methods fail because they decouple token reduction from semantic abstraction, scrubbing away fine-grained details without awareness of what objects are being merged. Funneling high-resolution data through such a narrow bottleneck creates a strict physical limit on pixel-level accuracy, severely hindering applications that require fine-grained spatial mapping (Rajat Koner et al.).

The Hidden Cost of Token Dropping

An alternative to pooling is token dropping, where models use attention weights to identify and discard "redundant" visual tokens early in the pipeline. However, this approach introduces a different kind of spatial disruption.

[ Continuous Image Grid ] ---> [ Naive Attention Pruning ] ---> [ Fragmented Spatial Map ]
(Tracks relative distances)     (Discards "low-attention"        (LLM loses geometric
                                 background tokens)               continuity & coordinates)

As Kazi Hasan Ibn Arif et al. show, aggressive pruning breaks the spatial continuity of an image. When background or intermediate tokens are dropped haphazardly, the geometric relationships between remaining tokens are severed, making it difficult for the model to track coordinates in layout-sensitive tasks like chart analysis.

This problem is magnified in sequential or dynamic contexts. Reinforcement learning frameworks designed to optimize token sparsity reveal that aggressive pruning often discards low-probability tokens (Feng Chen et al.). While these tokens may seem unimportant to an isolated attention layer, they are vital for maintaining an internal map of physical distances and object relationships across multiple frames. Without them, the model's internal representation of the scene becomes fragmented.

The Limits of Extreme Compression

The consequences of this information loss become starkest when compression is pushed to its logical extreme—such as compressing an entire image or frame into a single token or a small set of discrete codes.

When high-resolution images are forced into ultra-low token budgets, high-density spatial reasoning tasks like Optical Character Recognition (OCR) and small object detection fail completely (Shaolei Zhang et al.; Yuying Ge et al.). As Xubing Ye et al. observe, there is an inherent trade-off between the compression ratio and the retention of fine-grained features. General semantics (e.g., "a photo of a kitchen") survive extreme compression, but precise localization data is permanently lost.

This bottleneck has severe real-world implications. In robotic manipulation, standard token compression methods routinely lose track of small or distant target objects (Haifeng Huang et al.). When a transformer-based policy cannot resolve the exact boundaries of a cluttered visual field, its ability to generalize zero-shot actions to novel objects degrades. Similarly, in long-video modeling, aggressive hierarchical compression causes models to suffer from hallucinations, failing to maintain object identity or track subtle changes over time (Jiaheng Liu et al.; Xinhao Li et al.).

Intelligent Compression: Retaining the Coordinate Grid

To overcome these limitations, recent architectures focus on compressing smarter rather than harder, employing mechanisms that explicitly preserve spatial structure.

Language and Task Guidance

Instead of pruning tokens based solely on visual self-attention, models can use the language prompt as a filtering lens. By leveraging the user's instruction to determine token relevance, the system ensures that task-critical visual cues and spatial relationships are never discarded, even when pruning up to 90% of the image data (Yizheng Sun et al.; Bo Tong et al.).

Visual Anchors and Multi-Granularity Streams

Rather than treating all patches uniformly, frameworks like the one proposed by Haogeng Liu et al. identify "visual anchors"—specific tokens within the ViT that naturally aggregate surrounding information. This allows the model to condense data around structurally important focal points, preserving fine-grained details better than uniform pooling.

Mathematical Position Preservation

Perhaps the most direct solution to the coordinate-blending problem is to re-engineer how positional metadata is compressed.

Standard Pooling:  [Pos 1][Pos 2][Pos 3][Pos 4] ---> [ Averaged Vector ] (Coordinates lost)

Positional Preservation Embedding (PPE):
                   [Pos 1][Pos 2][Pos 3][Pos 4] ---> [ Compressed Vector ] 
                                                      (Dimensions 1-2: Semantics)
                                                      (Dimensions 3-6: Multi-position IDs)

As Mouxiao Huang et al. demonstrate with Positional Preservation Embedding (PPE), a single compressed token can be designed to encapsulate multiple original position IDs across its individual embedding dimensions. By packing the geometric layout directly into the compressed vector, the model maintains its internal spatial and temporal structure even at extreme reduction ratios, allowing the downstream LLM to successfully recover layout-sensitive details.

Key Foundational Work

VoCo-LLaMA: Towards Vision Compression with Large Language Models (2025) — contributed 3 concept elements to this direction.
Visual Anchors Are Strong Information Aggregators For Multimodal Large Language Model (2024) — contributed 1 concept elements to this direction.
DeCo: Decoupling Token Compression from Semantic Abstraction in Multimodal Large Language Models (2024) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_spatial_reasoning_degradation_7a3ca46c,
  title = {spatial reasoning degradation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/spatial-reasoning-degradation/rdr_7a3ca46c6e274627a8a60eee0ca70c22},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_spatial_reasoning_degradation_7a3ca46c_v1,
  title = {spatial reasoning degradation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/spatial-reasoning-degradation/rdr_7a3ca46c6e274627a8a60eee0ca70c22/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake