M1: Towards Scalable Test-Time Compute with Mamba Reasoning Models

Junxiong Wang, Wen-Ding Li, Daniele Paliotta, Daniel Ritter, Alexander M Rush, Tri Dao

TL;DR

Scaling test-time reasoning via long chains of thought or parallel sampling typically incurs heavy memory and compute bottlenecks in Transformers due to the key-value (KV) cache. M1 is a 3-billion-parameter hybrid Mamba-Transformer model that matches state-of-the-art reasoning models like DeepSeek-R1-Distill-Qwen-1.5B while achieving more than a 3× inference speedup at large batch sizes. Under fixed time budgets, this throughput advantage allows M1 to outperform standard Transformers on competition math via majority voting and long-sequence scaling.

What Problem Does This Paper Address?

Modern reasoning models (such as OpenAI's o1 or DeepSeek-R1) generate long chains of thought (CoT) and rely heavily on test-time compute strategies like self-consistency (sampling multiple solutions and majority voting). In standard Transformer architectures, long sequences and large generation batches are notoriously expensive:

  1. Memory footprints explode due to the KV cache, making large-batch decoding memory-bound.
  2. Quadratic attention costs slow down both generation rollouts during reinforcement learning (RL) training and inference serving.

While linear RNNs and State Space Models (SSMs) like Mamba offer linear time complexity and constant-size recurrent states, whether their expressiveness holds up during complex multi-step mathematical reasoning has been an open question. M1 tackles this challenge by demonstrating that hybrid Mamba architectures can master competition-level reasoning without sacrificing speed.

Methods

The authors construct M1-3B from Llama-3.2-3B-Instruct using a 28-layer hybrid architecture containing 6 interleaved attention layers and 22 Mamba layers (SSM state dimension 16, expanded across 192 groups). Training follows a three-stage recipe requiring under 50 billion tokens:

Pretrained Llama-3.2-3B 
   │  Stage 1: Cross-Architecture Distillation (Reverse KL)
   ▼
Base Hybrid Mamba Model
   │  Stage 2: Two-Phase SFT (Math foundations → Long CoT traces)
   ▼
SFT Reasoning Model
   │  Stage 3: RL with GRPO (No KL penalty + Entropy bonus, up to 32k tokens)
   ▼
M1-3B
  • Cross-Architecture Weight Transfer & Distillation: Attention layers are replaced by Mamba layers in one shot, reusing attention projection weights ($Q, K, V, O \to C, B, X, O$). To account for Grouped Query Attention (GQA) head reduction in Llama, two linear projection layers expand the inputs to increase state expressiveness. The student model is trained with reverse Kullback-Leibler (KL) divergence for mode-seeking distribution alignment.
  • Two-Phase Supervised Fine-Tuning (SFT): The model is first fine-tuned on general math solutions, then fine-tuned on 10 billion tokens of long reasoning traces (packed up to 24,576 tokens).
  • Reasoning Reinforcement Learning: The authors integrate Mamba into the VeRL framework using Group Relative Policy Optimization (GRPO) with rollouts up to 32k tokens. Crucially, they remove the KL divergence penalty (which destabilized training) and add an entropy bonus to encourage exploration.

Datasets

  • SFT Training:
  • Phase 1: OpenMathInstruct-2 (math instruction dataset).
  • Phase 2: A 10B-token CoT mixture comprising OpenR1-Math-220k, OpenThoughts-114k-math, ServiceNow-AI-R1-Distill, and Magpie-Reasoning-250K.
  • Evaluation Benchmarks: Evaluated on competitive math benchmarks: MATH500, AIME 2024, AIME 2025, AMC 2023, and OlympiadBench.

Key Insights and Ideas

  • Direct distillation from reasoning models fails: Attempting to distill DeepSeek-R1 traces directly into a raw hybrid model yielded poor results (only 38% on MATH500). The model first requires a solid foundation in basic math concepts (OpenMathInstruct) before it can absorb long-form reasoning behaviors.
  • Rollout generation is the RL bottleneck: In reasoning RL, rollout generation often takes over 3× longer than the backpropagation update. M1’s recurrent states eliminate large KV cache accesses, accelerating rollout generation and making RL on 24k–32k context windows practical.
  • Throughput translates directly into accuracy: Because hybrid Mamba inference is memory-efficient, M1 generates substantially more tokens per second at large batch sizes. Under a fixed wall-clock time budget, M1 achieves higher accuracy via majority voting (Maj@k) than same-sized or smaller Transformers because it can sample far more paths in parallel.

Key Results

  • Reasoning Parity: M1-3B performs on par with DeepSeek-R1-Distill-Qwen-1.5B and competitive 7B models across standard math benchmarks:
  • AIME 2024: 28.9% Pass@1 (50.5% Maj@32)
  • AIME 2025: 23.5% Pass@1 (34.6% Maj@32)
  • MATH500: 82.1% Pass@1 (91.8% Maj@32)
  • OlympiadBench: 47.3% Pass@1 (80.1% Maj@32)
  • Decoding Latency and Throughput: Compared to Llama-3.2-3B served on vLLM, M1 provides a 3× speedup at batch size 512 (decoding length 4,096) and remains 2.64× faster at long sequence lengths (~20k tokens).
  • Time-Constrained Scaling: M1 hits an optimal throughput of 15,169 tokens/sec on an H100 GPU compared to 7,263 tokens/sec for DeepSeek-R1-Distill-Qwen-1.5B, yielding higher accuracy per second of compute during self-consistency search.

Limitations and Future Work

  • Software Optimizations: M1’s attention layers were not yet fully merged into vLLM’s custom kernels. Integrating optimized hybrid kernels (such as NVIDIA’s hybrid Mamba kernel) could yield even larger speedups.
  • Teacher Model Dependency: The authors distilled from Llama-3.2 rather than Qwen2.5-Math-1.5B because the OpenMathInstruct dataset was closer to Llama's distribution (cross-entropy was 0.5 for Llama vs. >1.8 for Qwen). Creating Qwen-native distillation datasets remains future work.
  • Scale: The study focuses on 3B models; validating whether these distillation and RL training dynamics scale effectively to 7B–70B parameter hybrid models remains an open exploration area.

Summary updated

Abstract

Effective reasoning is crucial to solving complex mathematical problems.Recent large language models (LLMs) have boosted performance by scaling test-time computation through long chain-of-thought reasoning. However, transformer-based models are inherently limited in extending context length due to their quadratic computational complexity and linear memory requirements. In this paper, we introduce a novel hybrid linear RNN reasoning model, M1, built on the Mamba architecture, which allows memory-efficient inference. Our approach leverages a distillation process from existing reasoning models and is further enhanced through RL training. Experimental results on the AIME and MATH benchmarks show that M1 not only outperforms previous linear RNN models but also matches the performance of state-of-the-art Deepseek R1 distilled reasoning models at a similar scale. We also compare our generation speed with a highly performant general purpose inference engine, vLLM, and observe more than a 3x speedup compared to the same-sized transformer. With throughput speedup, we are able to achieve higher accuracy compared to DeepSeek R1 distilled transformer reasoning models under a fixed generation time budget using self-consistency voting. Overall, we introduce the hybrid Mamba reasoning model and provide a more effective approach to scaling test-time generation using self-consistency or long chain of thought reasoning.
BibTeX
@inproceedings{wang2025m,
  author = {Junxiong Wang and Wen-Ding Li and Daniele Paliotta and Daniel Ritter and Alexander M Rush and Tri Dao},
  title = {{M1: Towards Scalable Test-Time Compute with Mamba Reasoning Models}},
  booktitle = {ER @ NeurIPS},
  year = {2025},
  url = {https://openreview.net/forum?id=bOnhqVefxk},
}

Loading related content...

Figure