TL;DR
Visual State Space Models (SSMs) like VMamba offer linear-complexity alternatives to Vision Transformers, but their 4-directional scanning mechanism incurs significant spatial redundancy and latency bottlenecks. QuarterMap is a training-free, post-training activation pruning method that spatially downsamples feature maps to one-quarter size before the 2D scan and restores them using nearest-neighbor upsampling. Without retraining or architectural redesign, it boosts VMamba inference throughput by up to 11% on ImageNet-1K with under a 0.9% accuracy loss—avoiding the heavy merge/unmerge overheads that cripple methods like Token Merging (ToMe).
What Problem Does This Paper Address?
While Vision Transformers (ViTs) capture global context effectively, their self-attention mechanism scales quadratically with sequence length. Visual State Space Models (SSMs), particularly VMamba, replace attention with linear-time selective scanning. To handle 2D images, VMamba uses a 2D-Selective-Scan (SS2D) block that unfolds feature maps into four directional 1D sequences (cross-scan), runs selective scans in parallel, and merges them back (cross-merge).
However, selective scan kernels still account for nearly 18% of VMamba's runtime. More importantly, scanning across four directions causes adjacent tokens to accumulate redundant spatial information. Prior ViT acceleration methods like Token Merging (ToMe) require dynamic token-similarity calculations and unmerging operations, which add immense computational overhead in SSMs. Other SSM token pruning methods require full fine-tuning or retraining. QuarterMap addresses this challenge: How can we exploit spatial redundancy in visual SSMs to achieve real-time speedups at inference without retraining?
Methods
QuarterMap operates strictly during inference as a plug-and-play modification to the SS2D module:
- Layer & Block Selection: Pruning is applied periodically every $k$ blocks (setting $k=3$ offers the best Pareto efficiency) and skips the earliest layer, which encodes low-level spatial features sensitive to disruption.
- Pruning Stage: Before the cross-scan operation, QuarterMap subsamples the 2D activation map $x \in \mathbb{R}^{H \times W \times D}$ by retaining every alternate pixel along both spatial dimensions ($m=2, n=1$). This slashes the sequence length to $1/4$ ($⌈H/2⌉ \times ⌈W/2⌉$) before the 1D selective scan sequences are generated.
- SS2D Processing: Cross-scan, selective scan (Mamba recurrence), and cross-merge proceed on the quarter-sized feature map, significantly cutting linear projection and associative scan computation.
- Upsampling Stage: Immediately following cross-merge, original spatial dimensions are restored using nearest-neighbor interpolation. Nearest-neighbor was chosen over bilinear or bicubic interpolation because it introduces practically zero computational overhead while leveraging the property that adjacent pixels share closely aligned semantic states.
Datasets
The authors evaluate QuarterMap across multiple vision tasks:
- ImageNet-1K: Evaluated on the validation set for general visual recognition across Base, Small, and Tiny VMamba variants, alongside comparative baselines (ConvNeXt, DeiT, Swin, ViM).
- ADE20K: Used with the UperNet framework to evaluate semantic segmentation performance (measured via mean IoU and all-pixel accuracy).
- MedMNIST v2: Evaluated on four medical classification benchmarks—BloodMNIST, OrganMNIST, RetinaMNIST, and PathMNIST—using MedMamba-T.
No new datasets were introduced.
Key Insights and Ideas
- ToMe is a misfit for SSMs: While Token Merging works well in ViTs, applying ToMe to VMamba drops throughput by ~28–34% (a 0.66×–0.72× slowdown). ToMe incurs a massive 9.7 ms overhead from pairwise similarity calculation and unmerging, completely erasing the 0.7 ms kernel time saved. QuarterMap adds only 0.2 ms of overhead.
- SSM multi-directional scanning creates unique redundancy: The four-way scan repeatedly visits overlapping spatial neighborhoods. Subsampling before cross-scan eliminates redundant computation without losing critical structural features.
- Architectural specificity matters: QuarterMap fails drastically on CNNs (e.g., dropping ConvNeXt-B accuracy from 84.89% to 45.71%) because convolutions depend heavily on local continuity. It also triggers noticeable degradation in 1D-scanning SSMs like ViM (dropping from 80.40% to 71.00%), confirming that QuarterMap specifically capitalizes on the multi-directional scanning redundancy of 2D SSM backbones.
- Earlier layers must be preserved: Pruning Layer 1 causes severe accuracy drops (down to 78.75%), whereas pruning deeper layers (such as Layer 3) yields the highest speedups (up to 1.27×) with minimal loss, confirming that early SSM features require dense representations.
Key Results
- ImageNet-1K Classification:
- VMamba-B ($k=3$): Throughput improves by 1.11× (590 to 654 images/sec) with only a 0.86% accuracy drop (83.88% $\to$ 83.02%).
- VMamba-B ($k=2$): Throughput reaches 1.16× (682 images/sec) with a 1.30% drop.
- Scan kernel execution time drops from 1.4 ms down to 0.6 ms.
- ADE20K Segmentation: On VMamba-Base with UperNet, all-pixel accuracy drops by only 0.76% and mIoU drops by 1.75% (50.96% $\to$ 49.21%).
- Medical Imaging (MedMamba-T): Across BloodMNIST, OrganMNIST, RetinaMNIST, and PathMNIST, QuarterMap delivers a consistent 1.21× speedup (854 to ~1034 images/sec) with zero accuracy degradation (e.g., BloodMNIST accuracy marginally rose from 97.72% to 97.78%).
Limitations and Future Work
- Narrow Architecture Scope: QuarterMap is explicitly designed for models using VMamba-style 4-directional 2D selective scanning (like VMamba and MedMamba) and does not generalize to CNNs or 1D-scan SSMs.
- Trade-off on Small Backbones: Smaller models (like VMamba-Tiny) suffer slightly steeper accuracy drops (-1.10% at $k=3$, -2.59% at $k=2$) for modest throughput gains (1.05×–1.08×) compared to larger backbones.
- Future Directions: Combining QuarterMap with orthogonal post-training acceleration techniques (e.g., post-training quantization) could unlock compounding efficiency gains for resource-constrained edge deployments.
Summary updated