Vision state space models such as Vision Mamba handle a high-resolution image by flattening it into a long sequence and sweeping a compact running memory, the hidden state, across it. Their cost grows linearly with image size instead of quadratically as in self-attention. They are also entirely dense: every weight touches every pixel. Mixture-of-experts layers are the standard way to add parameters without adding per-pixel compute. In such a layer, a small learned "router" sends each token to one or two of many specialist sub-networks. Language models have already paired the two ideas, but vision cannot copy that recipe directly. A visual scan carries one continuous state across neighbouring pixels, often in four directions at once. Routing each pixel independently to a different expert makes the parameters that control how the state updates and forgets jump arbitrarily from one pixel to the next. It also scatters memory accesses in exactly the model family whose speed depends on tightly fused, contiguous kernels.
We propose to route contiguous segments of the scan path rather than individual pixels, and to share one routing decision across all scan directions that visit a pixel. The hidden state itself is never split and keeps flowing unbroken through the whole image. Only the projection weights that feed it and read from it are expert-specific, and they switch at segment boundaries. The claim has two parts. First, this design should give the capacity gains of sparse experts on classification, segmentation, and restoration at a fixed active compute budget. Second, it should do so where naive per-pixel expert routing fails to beat a dense model once wall-clock speed is counted.
Core Claim and Relation to Prior Work
The closest work is Routing Mamba. It turns Mamba's internal convolution, gate, and output projections into experts under a single router shared across those sub-layers. In one-dimensional language modeling, a 1.3B-active model matched a dense Mamba needing more than 2.3x the active parameters while keeping nearly 80% of dense throughput. We reuse its central method, projection-level experts under shared routing. Routing Mamba routes per token along a causal one-dimensional stream. It never faces non-causal two-dimensional grids, several scan directions per pixel, or dense spatial outputs. The other two language-side works, MoE-Mamba and BlackMamba, place experts only in the feed-forward layers between unconditional Mamba blocks. MoE-Mamba reached vanilla Mamba's perplexity in 42.5% of the training time. These papers motivate the proposal because they show sparse capacity pays off in state space models. They also define our strongest ablation, since feed-forward-only experts sidestep the state-continuity problem entirely.
On the vision side, V-MoE and MoE-ViE route image patches independently inside quadratic-attention Vision Transformers. Attention has no running state to fragment, so per-patch routing is harmless there. MoE-ViE also supplies a method we reuse: bias-based load balancing without an auxiliary loss, and grouped matrix-multiply Triton kernels that gave a >2.5x speedup over plain PyTorch. Dynamic Vision Mamba is the only conditional-computation work on Vision Mamba. It prunes tokens and skips whole scan directions per image, reporting a 35.2% FLOP cut for a 1.7% top-1 drop. It adds no capacity, and it had to replace deletion with "stopping updates" for segmentation. That result constrains the proposal: altering the token stream of a recurrent scan is harmful, and dense tasks need an intact grid. Spatial Mixture-of-Experts routes pixels by fixed coordinates in convolutional networks and ran slower than dense networks because of irregular memory access. It warns that fine-grained spatial routing without contiguity is a systems failure. PlainMamba motivates the choice of segment shape. It showed that keeping sequence neighbours as true two-dimensional neighbours, through a serpentine "zigzag" scan, improves representations. Contiguous segments of a zigzag path are therefore compact image regions.
The new element is not "MoE plus Mamba." It is a specific answer to the gap between per-token routing and continuous recurrent state. The answer has three parts: route at the level of contiguous segments, share each decision across scan directions, and keep the hidden state and its step-size parameters shared.
Mechanism
Take a PlainMamba-style non-hierarchical backbone. Partition each layer's zigzag sequence into segments of $L$ consecutive tokens, where $L$ ranges over 16, 64, and 256. A segment's router input is the mean of its token features plus a learned embedding of the segment's position. This gives content-driven routing with a weak coordinate prior, a middle ground between the fixed coordinate routing of Spatial MoE and the purely content-based routing of V-MoE.
The router picks the top-1 or top-2 of $E \in \{8, 16\}$ experts. Each expert holds its own input projection, gate projection, and output projection. Following Routing Mamba, the step-size projection, the short convolution, the state matrix, and the hidden state are shared and dense. This way, the forgetting dynamics that carry information across segment boundaries do not jump between experts. One routing decision per segment applies to every scan direction the block runs over that segment. It also applies to both the write side (projection into the state) and the read side (output projection). Because experts receive whole contiguous segments, dispatch is a block permutation of memory rather than a per-token gather. That permutation maps directly onto grouped matrix-multiply kernels of the MoE-ViE kind, and the selective-scan kernel runs unchanged over the full, unpermuted sequence.
Research Program
The work proceeds in three stages. The first stage builds the layer and its kernel. The target is at least 80% of the measured throughput of a dense model with equal active parameters at 224 and 1280 resolution, which mirrors Routing Mamba's reported retention. The second stage trains ImageNet-1K models at the PlainMamba-L1 and L2 active-compute scales, with total parameters raised 4–8x through experts. It then fine-tunes them for ADE20K segmentation and trains a MambaIR-style restoration variant. The third stage is a mechanistic study. We measure the norm of the hidden-state jump at routing boundaries under token-level versus segment-level routing, and relate it to accuracy. We also check whether experts specialize by region type, such as texture, edges, or flat background, the way V-MoE's experts did by layer.
Baselines
The naive reference is the dense PlainMamba at the same active parameter count, trained with the identical recipe. PlainMamba reports 81.6% ImageNet-1K top-1 for PlainMamba-L2 (25M parameters) and 49.1 mIoU on ADE20K. We retrain it ourselves so that data, epochs, augmentation, and tuning budget match exactly. Our tuning budget per model is a fixed small grid over learning rate and router settings, applied equally to all methods.
The strongest competitor on capacity is a dense PlainMamba widened or deepened until its measured training throughput equals our sparse model's. This is the comparison that decides whether sparsity is worth its complexity. Beating it means higher top-1 and mIoU at equal wall-clock training time and equal inference latency, not merely at equal FLOPs.
For context on absolute accuracy, we report the stronger published backbones without claiming parity of recipe. PRIS-Mamba-T reports 84.5% top-1 (with VMamba-T at 82.6%), and Adventurer-Base reports 84.3% at 216 images/s. Adventurer uses a single scan direction per layer, so we additionally port segment routing onto it. This tests whether the gain survives in the fastest known design.
The key ablations isolate each ingredient in turn. We compare token-level routing against segment-level routing at matched expert count. We compare one routing decision shared across directions against separate routers per direction. We compare expert projections against experts only in the feed-forward layers, in the style of MoE-Mamba. Finally, we test making the step-size projection expert-specific, which should break state continuity if our hypothesis is right.
For restoration, the file gives only relative gains. MambaIR reports beating SwinIR by up to 0.45 dB on Manga109 ×2 super-resolution and 0.48 dB on Urban100 σ=50 denoising. No absolute PSNR was found on Lacuna for MambaIR, so we retrain MambaIR as the dense restoration reference.
Why not just…?
Why not just scale the dense Vision Mamba? This is the most serious threat. Dynamic Vision Mamba notes that FLOP savings in Mamba do not translate linearly into speed, so sparse capacity might lose once latency is counted. The throughput-matched dense baseline settles this directly. If the widened dense model ties us at equal wall-clock time, we will report that sparse experts do not pay in vision state space models at this scale. That would be a useful negative result, because the language-side evidence from MoE-Mamba and Routing Mamba suggests they should.
Why not just put experts in the feed-forward layers, as MoE-Mamba does? This avoids the continuity problem altogether and is the simplest credible design. In one dimension, Routing Mamba found that projection-level experts under shared routing beat feed-forward-level experts, but that finding has never been tested on images. We run the feed-forward-only variant at matched active and total parameters. If it matches segment routing, the proposal reduces to that simpler recipe, and we will say so.
Why not just port Routing Mamba with per-token routing? Per-token routing is the default, and it may be fine. The segment-size sweep includes $L=1$, which is exactly this port. We predict it will show larger state jumps and a lower accuracy-per-latency trade-off. That prediction rests on the fragmentation effects that PlainMamba and Dynamic Vision Mamba observed. If $L=1$ wins, the segment idea is refuted.
Why not just prune tokens or skip scans? Dynamic Vision Mamba shows pruning trades accuracy for speed and needs workarounds on dense tasks. It cannot add capacity, so it answers a different question. We treat it as complementary rather than as a competitor on accuracy, and we report it as an efficiency reference point.
Proposed Experiments and Decisive Checks
We train on ImageNet-1K at 224 resolution, then fine-tune on ADE20K. Restoration uses Manga109 and Urban100 for super-resolution and denoising, with training and evaluation following MambaIR's settings. Metrics are top-1 accuracy, mIoU, PSNR and SSIM, measured training and inference throughput, peak memory at 1280×1280, and expert load entropy.
The proposal succeeds if three conditions hold. First, the segment-routed model beats the throughput-matched dense PlainMamba on both ImageNet and ADE20K. Second, it keeps at least 80% of equal-active-parameter dense throughput at 1280 resolution. Third, the ablations show two things: segment routing outperforms per-token routing, and making the step-size projection expert-specific hurts. That last pair of results would directly tie the gain to preserving state continuity. The proposal fails if the feed-forward-only variant or the widened dense model ties us under matched wall-clock budgets.
References
- Routing Mamba: Scaling State Space Models with Mixture-of-Experts Projection
- MoE-Mamba: Efficient Selective State Space Models with Mixture of Experts
- BlackMamba: Mixture of Experts for State-Space Models
- Scaling Vision with Sparse Mixture of Experts
- MoE-ViE: Mixture of Experts Vision Encoder for Efficient Image and Video Understanding
- Dynamic Vision Mamba
- Spatial Mixture-of-Experts
- PlainMamba: Improving Non-Hierarchical Mamba in Visual Recognition
- Partial Ring Scan: Revisiting Scan Order in Vision State Space Models
- Adventurer: Optimizing Vision Mamba Architecture Designs for Efficiency
- MambaIR: A Simple Baseline for Image Restoration with State-Space Model