MambaOut: Do We Really Need Mamba for Vision?

Weihao Yu, Xinchao Wang

Introduction

The "Mamba" architecture, based on State Space Models (SSMs), has gained significant traction in AI for its promise of linear complexity in processing long sequences. While highly successful in Natural Language Processing (NLP), its application to computer vision has yielded mixed results. This paper investigates a provocative question: Is the Mamba architecture actually necessary for vision tasks? By developing MambaOut, a model that strips the core SSM component from Mamba, the authors demonstrate that while Mamba may be overkill for standard image classification, its unique "RNN-like" memory remains a powerful asset for complex, high-resolution tasks like object detection and segmentation.

The "Mamba-Ready" Hypothesis

Most "Visual Mamba" models aim to replace computationally expensive Transformers. However, these often underperform compared to classic Convolutional Neural Networks (CNNs). The authors argue that Mamba's SSM core is specifically designed for two characteristics:

  1. Long Sequences: Handling thousands of tokens without the quadratic memory explosion seen in Transformers.
  2. Autoregressive/Causal Processing: Predicting subsequent tokens based on previous ones, similar to traditional RNNs.

Crucially, many vision tasks, such as ImageNet classification, do not inherently possess these characteristics.

Methodology: From Mamba to MambaOut

To isolate the contribution of the SSM component, the authors introduced MambaOut. A standard Mamba block is essentially a Gated CNN block with an integrated SSM layer. MambaOut simplifies this by removing the SSM, resulting in a stack of Gated CNN blocks that utilize simple 7x7 depthwise convolutions as the token mixer.

Architecture comparison and ImageNet performance.

By comparing MambaOut against state-of-the-art Visual Mamba models like Vim, VMamba, and PlainMamba, the researchers could determine where the SSM provides a genuine advantage and where it acts as unnecessary overhead.

Key Insights: Memory and Sequence Length

The study introduces a mathematical metric ($L > 6D$, where $L$ is sequence length and $D$ is model dimension) to define a "long sequence." Standard ImageNet classification (typically 196 tokens) falls into the "short sequence" category. In such cases, the "lossy" memory of Mamba's RNN-like structure is less effective than the "lossless" memory of attention mechanisms or the local focus of convolutions.

Comparison of memory mechanisms: Causal attention vs. RNN-like.

Furthermore, while NLP is inherently causal (left-to-right), vision is "fully visible." Forcing an image into a causal processing sequence—a constraint of many SSM implementations—can actually degrade performance on tasks where the entire spatial context is available simultaneously.

Experimental Results

The models were evaluated on ImageNet-1K (classification), MS COCO (detection/segmentation), and ADE20K (semantic segmentation).

  • ImageNet Performance: MambaOut-Small achieved 84.1% top-1 accuracy, outperforming all visual Mamba models of comparable size. For instance, it surpassed LocalVMamba-S by 0.4% while requiring only 79% of the computation (MACs).
  • The Long-Sequence Turning Point: On COCO and ADE20K, which involve much higher resolutions (approx. 4,000 tokens), MambaOut lagged behind the best Visual Mamba models. This performance gap confirms that Mamba’s SSM provides a unique advantage specifically for long-sequence vision tasks.

Conclusion

MambaOut serves as a "natural baseline" that highlights the specific strengths of SSMs. The findings suggest that researchers should avoid "Mamba-fying" every vision task. Instead, future work should focus on hybrid architectures that deploy SSMs strategically where their long-sequence handling and RNN-like memory provide a clear empirical edge, such as in high-resolution detection or specialized domains like medical imaging.

Summary updated

Abstract

Mamba, an architecture with RNN-like token mixer of state space model (SSM), was recently introduced to address the quadratic complexity of the attention mechanism and subsequently applied to vision tasks. Nevertheless, the performance of Mamba for vision is often underwhelming when compared with convolutional and attention-based models. In this paper, we delve into the essence of Mamba, and conceptually conclude that Mamba is ideally suited for tasks with long-sequence and autoregressive characteristics. For vision tasks, as image classification does not align with either characteristic, we hypothesize that Mamba is not necessary for this task; Detection and segmentation tasks are also not autoregressive, yet they adhere to the long-sequence characteristic, so we believe it is still worthwhile to explore Mamba's potential for these tasks. To empirically verify our hypotheses, we construct a series of models named MambaOut through stacking Mamba blocks while removing their core token mixer, SSM. Experimental results strongly support our hypotheses. Specifically, our MambaOut model surpasses all visual Mamba models on ImageNet image classification, indicating that Mamba is indeed unnecessary for this task. As for detection and segmentation, MambaOut cannot match the performance of state-of-the-art visual Mamba models, demonstrating the potential of Mamba for long-sequence visual tasks. The code is available at https://github.com/yuweihao/MambaOut
BibTeX
@inproceedings{yu2025mambaout,
  author = {Weihao Yu and Xinchao Wang},
  title = {{MambaOut: Do We Really Need Mamba for Vision?}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  doi = {10.1109/cvpr52734.2025.00423},
}
Figure