MambaOut: Do We Really Need Mamba for Vision?

Weihao Yu, Xinchao Wang

Abstract

Mamba, an architecture with RNN-like token mixer of state space model (SSM), was recently introduced to address the quadratic complexity of the attention mechanism and subsequently applied to vision tasks. Nevertheless, the performance of Mamba for vision is often underwhelming when compared with convolutional and attention-based models. In this paper, we delve into the essence of Mamba, and conceptually conclude that Mamba is ideally suited for tasks with long-sequence and autoregressive characteristics. For vision tasks, as image classification does not align with either characteristic, we hypothesize that Mamba is not necessary for this task; Detection and segmentation tasks are also not autoregressive, yet they adhere to the long-sequence characteristic, so we believe it is still worthwhile to explore Mamba's potential for these tasks. To empirically verify our hypotheses, we construct a series of models named MambaOut through stacking Mamba blocks while removing their core token mixer, SSM. Experimental results strongly support our hypotheses. Specifically, our MambaOut model surpasses all visual Mamba models on ImageNet image classification, indicating that Mamba is indeed unnecessary for this task. As for detection and segmentation, MambaOut cannot match the performance of state-of-the-art visual Mamba models, demonstrating the potential of Mamba for long-sequence visual tasks.

Introduction

In the wake of the Mamba architecture’s success in Natural Language Processing (NLP), researchers have rapidly adapted it for Computer Vision. Mamba’s primary appeal is its linear complexity—unlike Transformers, which scale quadratically with sequence length, Mamba maintains efficiency. However, "Visual Mamba" models frequently underperform compared to state-of-the-art CNNs or Transformers. This paper addresses a fundamental question: Do we actually need the specific mechanics of Mamba for vision?

The authors hypothesize that Mamba is ideally suited for tasks with long-sequence and autoregressive (causal) characteristics. If a vision task lacks these traits, Mamba may be unnecessary overhead.

Methods: The MambaOut Architecture

To test their hypothesis, the authors developed MambaOut. The core idea is simple: take a standard Mamba block and remove the State Space Model (SSM)—the defining "recurrent" component of Mamba. What remains is essentially a Gated CNN block.

Figure 1: (a) Comparison between the Gated CNN block used in MambaOut and the standard Mamba block. (b) Performance comparison on ImageNet-1K showing MambaOut surpassing various Visual Mamba models.

MambaOut follows a hierarchical structure similar to ResNet, using four stages. It employs simple 7x7 depthwise convolutions as the primary mechanism for mixing information between pixels, adhering to the principle of simplicity to establish a clear baseline.

Conceptual Analysis of Vision Tasks

The authors categorize vision tasks based on two key characteristics:

  1. Sequence Length: ImageNet classification (224x224) results in only 196 tokens. The authors demonstrate that for such "short-sequence" tasks, the efficiency of Mamba does not provide a significant advantage over Transformers or CNNs.
  2. Autoregressive vs. Understanding: Mamba is inherently causal, meaning it processes information in a sequence where a token only "sees" preceding information. Vision, however, is typically an "understanding" task where the model can observe the entire image simultaneously. Forcing causality on vision tasks can actually degrade performance.
Figure 5: Conceptual comparison of memory mechanisms. (a) Causal attention accumulates keys and values. (b) RNN-like (SSM) models use a compressed hidden state $h_t$.

Experimental Results

The models were evaluated on ImageNet-1K (classification), COCO (detection), and ADE20K (semantic segmentation).

  • ImageNet Classification: MambaOut consistently outperformed all existing Visual Mamba models. For example, MambaOut-Small achieved 84.1% accuracy, surpassing VMamba-S (83.5%) while maintaining higher efficiency. This supports the hypothesis that Mamba is unnecessary for standard image classification.
  • Detection and Segmentation: In these tasks, which involve much higher resolutions and thus longer sequences, the results shifted. MambaOut could not match the performance of the best Visual Mamba models. This confirms that Mamba’s ability to handle long-range dependencies efficiently provides a genuine boost for high-resolution visual tasks.

Conclusion

MambaOut serves as a critical baseline, refining the scope of Mamba's utility in computer vision. While it demonstrates that Mamba is "overkill" for simple image classification, it validates Mamba's potential for complex, high-resolution tasks. Future research in Visual Mamba should pivot away from classification benchmarks and focus on long-sequence visual tasks where SSMs offer a unique architectural advantage.

Summary updated

Abstract

Mamba, an architecture with RNN-like token mixer of state space model (SSM), was recently introduced to address the quadratic complexity of the attention mechanism and subsequently applied to vision tasks. Nevertheless, the performance of Mamba for vision is often underwhelming when compared with convolutional and attention-based models. In this paper, we delve into the essence of Mamba, and conceptually conclude that Mamba is ideally suited for tasks with long-sequence and autoregressive characteristics. For vision tasks, as image classification does not align with either characteristic, we hypothesize that Mamba is not necessary for this task; Detection and segmentation tasks are also not autoregressive, yet they adhere to the long-sequence characteristic, so we believe it is still worthwhile to explore Mamba's potential for these tasks. To empirically verify our hypotheses, we construct a series of models named MambaOut through stacking Mamba blocks while removing their core token mixer, SSM. Experimental results strongly support our hypotheses. Specifically, our MambaOut model surpasses all visual Mamba models on ImageNet image classification, indicating that Mamba is indeed unnecessary for this task. As for detection and segmentation, MambaOut cannot match the performance of state-of-the-art visual Mamba models, demonstrating the potential of Mamba for long-sequence visual tasks. The code is available at https://github.com/yuweihao/MambaOut
BibTeX
@inproceedings{yu2025mambaout,
  author = {Weihao Yu and Xinchao Wang},
  title = {{MambaOut: Do We Really Need Mamba for Vision?}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  doi = {10.1109/cvpr52734.2025.00423},
}
Figure