ToSA: Token Merging with Spatial Awareness

Hsiang-Wei Huang, Wenhao Chai, Kuang-Ming Chen, Cheng-Yen Yang, Jenq-Neng Hwang

Abstract

Vision Transformers (ViTs) are powerful but computationally expensive due to the quadratic complexity of self-attention. While token merging techniques like ToMe reduce this cost by combining similar tokens, they often fail to preserve spatial structure, leading to poor performance on tasks requiring precise counting and spatial reasoning. We present ToSA (Token Merging with Spatial Awareness), a training-free method that incorporates depth information into the merging process. By using a fused similarity score that accounts for both visual features and 3D spatial proximity, ToSA significantly improves performance on spatial benchmarks with negligible computational overhead.

Introduction: The Spatial Blind Spot in Token Merging

The efficiency of Vision Transformers (ViTs) is a critical bottleneck for their deployment in real-time systems, particularly in Vision-Language Models (VLMs). Token merging has emerged as a popular solution, where redundant patches are combined to reduce the sequence length. However, existing methods like ToMe rely almost exclusively on visual feature similarity.

In the initial layers of a transformer, these features are often low-level, such as edges or textures. This reliance causes the model to merge tokens from distinct objects that happen to share similar visual properties, effectively "smearing" the spatial layout of the scene. This lack of spatial awareness is particularly detrimental for embodied AI and robotics, where understanding the exact number and position of objects is paramount.

Methodology: Adding a Sense of Space

ToSA addresses this limitation by introducing a spatial prior into the token merging process. The method consists of three core components:

1. Pseudo Spatial Tokens

For every image patch, ToSA generates a triplet of coordinates $(x, y, z)$. The $x$ and $y$ values represent the standard 2D grid position, while $z$ represents the relative depth. This depth information is typically sourced from a pre-trained depth estimation model, such as Depth-Anything v2, or from native RGB-D sensors in robotic applications.

A depth map visualization, which ToSA utilizes to provide the 'z' coordinate for its 3D spatial priors.

2. Fused Similarity Scoring

Instead of deciding which tokens to merge based solely on visual appearance ($S_{visual}$), ToSA calculates a fused similarity score ($S_{fused}$): $$S_{fused} = \alpha S_{visual} + (1 - \alpha) S_{spatial}$$ where $S_{spatial}$ measures the proximity of tokens in 3D space. This ensures that tokens are only merged if they are both visually similar and spatially close.

3. Dynamic Alpha Schedule

The importance of spatial information varies across the transformer's depth. In early layers, where visual features are still being formed, spatial proximity is a more reliable guide for grouping. ToSA employs a linear "increase schedule" for $\alpha$, starting with a high weight for spatial similarity in early layers and gradually shifting toward visual/semantic similarity in deeper layers.

Results and Key Insights

ToSA was evaluated across several rigorous benchmarks, including SpatialBench and VQAv2-Counting. The results demonstrate that spatial awareness is a game-changer for token reduction:

  • Superior Counting Accuracy: On the SpatialBench "Counting" task, ToSA outperformed ToMe by +14.0% at a 90% token reduction rate. On VQAv2-Counting, accuracy rose from 70.2% to 77.7%.
  • Enhanced Spatial Reasoning: The model showed a +10.0% improvement in "Existence" tasks, proving that maintaining spatial boundaries prevents the model from "losing" objects during merging.
  • Efficiency: ToSA maintains high throughput, achieving approximately 33.5 images/sec. The computational overhead of calculating 3D similarity is minimal, resulting in less than a 0.6% speed decrease compared to the spatially-unaware ToMe.

Conclusion

ToSA demonstrates that a simple spatial prior can resolve the "blind spot" of existing token merging techniques. By making ViTs spatially aware, we can achieve significant speedups without sacrificing the structural integrity of the visual scene. As a training-free, plug-and-play module, ToSA is immediately applicable to state-of-the-art VLMs, particularly in domains like robotics and autonomous driving where 3D context is readily available and spatial precision is non-negotiable.

Summary updated

BibTeX
@inproceedings{huang2025tosa,
  author = {Hsiang-Wei Huang and Wenhao Chai and Kuang-Ming Chen and Cheng-Yen Yang and Jenq-Neng Hwang},
  title = {{ToSA: Token Merging with Spatial Awareness}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2025},
  doi = {10.1109/iros60139.2025.11245824},
}
Figure