VSSD: Vision Mamba with Non-Casual State Space Duality

Yuheng Shi, Minjing Dong, Mingjia Li, Chang Xu

Abstract

Vision transformers have significantly advanced the field of computer vision, offering robust modeling capabilities and global receptive field. However, their high computational demands limit their applicability in processing long sequences. To tackle this issue, State Space Models (SSMs) have gained prominence in vision tasks as they offer linear computational complexity. Recently, State Space Duality (SSD), an improved variant of SSMs, was introduced in Mamba2 to enhance model performance and efficiency. However, the inherent causal nature of SSD/SSMs restricts their applications in non-causal vision tasks. To address this limitation, we introduce Visual State Space Duality (VSSD) model, which has a non-causal format of SSD. Specifically, we propose to discard the magnitude of interactions between the hidden state and tokens while preserving their relative weights, which relieves the dependencies of token contribution on previous tokens. Together with the involvement of multi-scan strategies, we show that the scanning results can be integrated to achieve non-causality, which not only improves the performance of SSD in vision tasks but also enhances its efficiency. We conduct extensive experiments on various benchmarks including image classification, detection, and segmentation, where VSSD surpasses existing state-of-the-art SSM-based models.

Introduction

Vision Transformers (ViTs) have set the standard for computer vision but face a significant bottleneck: quadratic computational complexity. As image resolution increases, the memory and time required grow exponentially. State Space Models (SSMs), particularly the Mamba architecture, have emerged as a linear-complexity alternative.

However, SSMs were originally designed for 1D causal sequences (like text), where a token only interacts with preceding elements. Applying this to 2D images is inherently problematic. Previous "Vision Mamba" attempts relied on complex multi-scan strategies—scanning images in four or eight directions—to approximate 2D spatial relationships. These strategies are computationally expensive and still fundamentally treat 2D data as a series of 1D causal sequences. VSSD addresses this by reformulating the Mamba2 "SSD" block into a natively non-causal and position-independent format.

VSSD Motivation and Performance: (a) illustrates how causality restricts token interaction in standard SSD, while (b) and (c) show VSSD's superior accuracy-efficiency trade-offs.

Non-Causal State Space Duality (NC-SSD)

The core innovation of VSSD is the Non-Causal State Space Duality (NC-SSD) block. The authors identified that the recurrent, causal nature of standard SSMs is a hindrance for static images. By mathematically transforming the SSD formulation, they removed the causal mask that restricts token interactions.

As shown in the architectural comparison below, standard SSD relies on a lower-triangular matrix $M$, enforcing that tokens only "see" the past. NC-SSD discards the magnitude of interactions between hidden states and tokens while preserving relative weights, resulting in a dense interaction matrix. This allows the model to derive a single global hidden state $H$ shared by all tokens, enabling full parallelization during training and inference.

Comparison of standard SSD (left) and the proposed NC-SSD (right). NC-SSD removes the causal constraint in matrix M, allowing for a global hidden state H and parallel computation.

Hybrid Architecture

VSSD utilizes a hierarchical four-stage structure. The first three stages employ the efficient NC-SSD blocks for large feature maps, while the final stage uses standard Multi-head Self-Attention (MSA) to capture high-level global dependencies in condensed features. This hybrid approach balances the linear efficiency of SSMs with the modeling power of attention.

Experimental Results

VSSD was evaluated across major benchmarks, consistently outperforming both traditional Transformers and previous SSM-based models:

  • ImageNet-1K Classification: VSSD-Tiny achieves 83.7% top-1 accuracy, a significant improvement over VMamba-T (82.5%) and Swin-T (81.3%) at similar computational scales.
  • Object Detection & Segmentation: On MS COCO, VSSD-T reached 46.9 box AP, a +4.2 gain over Swin-T. On ADE20K semantic segmentation, it achieved 47.9 mIoU, outperforming Swin-T by +3.5.
  • Efficiency Gains: By eliminating the need for multi-directional scans and enabling full parallelism, VSSD trains nearly 50% faster than bidirectional SSM models and offers a superior accuracy-latency Pareto front compared to ConvNeXt and Swin.

Conclusion

VSSD demonstrates that causality is an obstacle, rather than a feature, for vision-centric State Space Models. By reformulating SSD into a non-causal format, VSSD achieves state-of-the-art performance with significantly higher efficiency. Future work will explore scaling VSSD to larger parameter counts and testing its robustness on even more diverse datasets like ImageNet-22K.

Summary updated

Abstract

Vision transformers have significantly advanced the field of computer vision, offering robust modeling capabilities and global receptive field. However, their high computational demands limit their applicability in processing long sequences. To tackle this issue, State Space Models (SSMs) have gained prominence in vision tasks as they offer linear computational complexity. Recently, State Space Duality (SSD), an improved variant of SSMs, was introduced in Mamba2 to enhance model performance and efficiency. However, the inherent causal nature of SSD/SSMs restricts their applications in non-causal vision tasks. To address this limitation, we introduce Visual State Space Duality (VSSD) model, which has a non-causal format of SSD. Specifically, we propose to discard the magnitude of interactions between the hidden state and tokens while preserving their relative weights, which relieves the dependencies of token contribution on previous tokens. Together with the involvement of multi-scan strategies, we show that the scanning results can be integrated to achieve non-causality, which not only improves the performance of SSD in vision tasks but also enhances its efficiency. We conduct extensive experiments on various benchmarks including image classification, detection, and segmentation, where VSSD surpasses existing state-of-the-art SSM-based models. Code and weights are available at \url{https://github.com/YuHengsss/VSSD}.
BibTeX
@inproceedings{shi2025vssd,
  author = {Yuheng Shi and Minjing Dong and Mingjia Li and Chang Xu},
  title = {{VSSD: Vision Mamba with Non-Casual State Space Duality}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2025},
}
Figure