QuarterMap: Efficient Post-Training Token Pruning for Visual State Space Models

Tien-Yu Chi, Hung-Yueh Chiang, Diana Marculescu, Kai-Chiang Wu

Abstract

State space models (SSMs) reduce the quadratic complexity of transformers by leveraging linear recurrence. Recently, VMamba has emerged as a strong SSM-based vision backbone, yet remains bottlenecked by spatial redundancy in its four-directional scan. We propose QuarterMap, a post-training activation pruning method that removes redundant spatial activations before scanning and restores dimensions via nearest-neighbor upsampling. Our method improves throughput without retraining. On ImageNet-1K, QuarterMap achieves up to 11% speedup on VMamba with less than 0.9% accuracy drop, and yields similar gains on ADE20K segmentation. Beyond VMamba, we validate QuarterMap on MedMamba, a domain-specific model that shares the same four-directional scanning structure, where it consistently improves throughput while preserving accuracy across multiple medical imaging tasks. Compared to token merging methods like ToMe, QuarterMap is tailored for SSMs and avoids costly merge-unmerge operations. Our method offers a plug-and-play tool for deployment-time efficiency without compromising transferability.

Introduction

Vision Transformers (ViTs) have revolutionized computer vision but suffer from quadratic computational complexity relative to input resolution. State Space Models (SSMs), such as the VMamba architecture, offer a linear-complexity alternative. However, VMamba's core mechanism, the 2D-Selective-Scan (SS2D), introduces its own overhead. By scanning feature maps in four directions to capture spatial context, it processes significant amounts of redundant information, as adjacent pixels in natural images often share high mutual information.

Existing optimization techniques like Token Merging (ToMe) are ill-suited for SSMs. The complex bipartite matching logic of ToMe introduces significant latency overhead that often outweighs the computational savings from token reduction in the SSM kernel.

Methodology: The QuarterMap Pipeline

QuarterMap is a training-free, post-training pruning method designed to exploit the spatial redundancy in SSMs. It operates as a lightweight "plug-and-play" module within the VSS blocks of the model.

Figure 2: Overview of the QuarterMap pipeline compared to the standard VSS block. QuarterMap prunes activations before the cross-scan operation and restores them via upsampling.

The process consists of three main stages:

  1. Block Selection: Pruning is applied selectively. Early layers, which encode fundamental low-level features, are preserved. QuarterMap is typically integrated into every $k$-th block (e.g., $k=2$ or $k=3$) in the deeper, more redundant stages of the network.
  2. Pruning Stage: Before the computationally expensive cross-scan operation, the feature map is downsampled. In its standard configuration, it retains one out of every four pixels, effectively reducing the sequence length by 75%.
  3. Upsampling Stage: Following the scan and merge operations, the original spatial dimensions are restored using nearest-neighbor interpolation. This simple restoration is effective because the four-directional scan in SSMs naturally aggregates contextual information from neighbors into the remaining tokens.

Experimental Results

The authors evaluated QuarterMap on ImageNet-1K for classification, ADE20K for segmentation, and MedMNIST for medical imaging.

Figure 1: Throughput vs. Top-1 Accuracy on ImageNet-1K. QuarterMap (QM) consistently improves throughput with minimal accuracy loss, whereas ToMe on VMamba-Base significantly reduces throughput due to overhead.

Efficiency and Performance

On the VMamba-Base model, QuarterMap achieved a 1.11× throughput increase with a negligible 0.86% drop in Top-1 accuracy. Crucially, while ToMe resulted in a throughput decrease (0.72×) due to its 9.7ms overhead, QuarterMap's overhead was measured at just 0.2ms.

Domain Generalization

In medical imaging tasks using MedMamba, QuarterMap demonstrated even greater efficacy. It provided a 1.21× throughput boost (from 854 to 1034 images/sec) with zero loss in accuracy across multiple datasets, including BloodMNIST and OrganMNIST. This suggests that domain-specific models with high spatial redundancy are ideal candidates for this method.

Conclusion

QuarterMap provides an efficient, training-free solution for accelerating Visual SSMs. By leveraging the inherent properties of the 2D-Selective-Scan, it removes spatial redundancy with minimal computational overhead. While highly effective for 2D-scanning architectures like VMamba and MedMamba, its application to 1D-scanning SSMs or standard CNNs remains a challenge due to different redundancy patterns and kernel requirements. Future work may explore adaptive pruning strategies to further optimize the trade-off between efficiency and precision.

Summary updated

Abstract

State space models (SSMs) reduce the quadratic complexity of transformers by leveraging linear recurrence. Recently, VMamba has emerged as a strong SSM-based vision backbone, yet remains bottlenecked by spatial redundancy in its four-directional scan. We propose QuarterMap, a post-training activation pruning method that removes redundant spatial activations before scanning and restores dimensions via nearest-neighbor upsampling. Our method improves throughput without retraining. On ImageNet-1K, QuarterMap achieves up to 11% speedup on VMamba with less than 0.9% accuracy drop, and yields similar gains on ADE20K segmentation. Beyond VMamba, we validate QuarterMap on MedMamba, a domain-specific model that shares the same four-directional scanning structure, where it consistently improves throughput while preserving accuracy across multiple medical imaging tasks. Compared to token merging methods like ToMe, QuarterMap is tailored for SSMs and avoids costly merge-unmerge operations. Our method offers a plug-and-play tool for deployment-time efficiency without compromising transferability.
BibTeX
@inproceedings{chi2025quartermap,
  author = {Tien-Yu Chi and Hung-Yueh Chiang and Diana Marculescu and Kai-Chiang Wu},
  title = {{QuarterMap: Efficient Post-Training Token Pruning for Visual State Space Models}},
  booktitle = {ES-FoMo-III @ ICML},
  year = {2025},
  url = {https://openreview.net/forum?id=pGJB9zGGwC},
}
Figure