TL;DR
Scaling test-time reasoning via long chains of thought or parallel sampling typically incurs heavy memory and compute bottlenecks in Transformers due to the key-value (KV) cache. M1 is a 3-billion-parameter hybrid Mamba-Transformer model that matches state-of-the-art reasoning models like DeepSeek-R1-Distill-Qwen-1.5B while achieving more than a 3× inference speedup at large batch sizes. Under fixed time budgets, this throughput advantage allows M1 to outperform standard Transformers on competition math via majority voting and long-sequence scaling.
What Problem Does This Paper Address?
Modern reasoning models (such as OpenAI's o1 or DeepSeek-R1) generate long chains of thought (CoT) and rely heavily on test-time compute strategies like self-consistency (sampling multiple solutions and majority voting). In standard Transformer architectures, long sequences and large generation batches are notoriously expensive:
- Memory footprints explode due to the KV cache, making large-batch decoding memory-bound.
- Quadratic attention costs slow down both generation rollouts during reinforcement learning (RL) training and inference serving.
While linear RNNs and State Space Models (SSMs) like Mamba offer linear time complexity and constant-size recurrent states, whether their expressiveness holds up during complex multi-step mathematical reasoning has been an open question. M1 tackles this challenge by demonstrating that hybrid Mamba architectures can master competition-level reasoning without sacrificing speed.
Methods
The authors construct M1-3B from Llama-3.2-3B-Instruct using a 28-layer hybrid architecture containing 6 interleaved attention layers and 22 Mamba layers (SSM state dimension 16, expanded across 192 groups). Training follows a three-stage recipe requiring under 50 billion tokens:
Pretrained Llama-3.2-3B
│ Stage 1: Cross-Architecture Distillation (Reverse KL)
▼
Base Hybrid Mamba Model
│ Stage 2: Two-Phase SFT (Math foundations → Long CoT traces)
▼
SFT Reasoning Model
│ Stage 3: RL with GRPO (No KL penalty + Entropy bonus, up to 32k tokens)
▼
M1-3B
- Cross-Architecture Weight Transfer & Distillation: Attention layers are replaced by Mamba layers in one shot, reusing attention projection weights ($Q, K, V, O \to C, B, X, O$). To account for Grouped Query Attention (GQA) head reduction in Llama, two linear projection layers expand the inputs to increase state expressiveness. The student model is trained with reverse Kullback-Leibler (KL) divergence for mode-seeking distribution alignment.
- Two-Phase Supervised Fine-Tuning (SFT): The model is first fine-tuned on general math solutions, then fine-tuned on 10 billion tokens of long reasoning traces (packed up to 24,576 tokens).
- Reasoning Reinforcement Learning: The authors integrate Mamba into the VeRL framework using Group Relative Policy Optimization (GRPO) with rollouts up to 32k tokens. Crucially, they remove the KL divergence penalty (which destabilized training) and add an entropy bonus to encourage exploration.
Datasets
- SFT Training:
- Phase 1:
OpenMathInstruct-2(math instruction dataset). - Phase 2: A 10B-token CoT mixture comprising
OpenR1-Math-220k,OpenThoughts-114k-math,ServiceNow-AI-R1-Distill, andMagpie-Reasoning-250K. - Evaluation Benchmarks: Evaluated on competitive math benchmarks: MATH500, AIME 2024, AIME 2025, AMC 2023, and OlympiadBench.
Key Insights and Ideas
- Direct distillation from reasoning models fails: Attempting to distill DeepSeek-R1 traces directly into a raw hybrid model yielded poor results (only 38% on MATH500). The model first requires a solid foundation in basic math concepts (
OpenMathInstruct) before it can absorb long-form reasoning behaviors. - Rollout generation is the RL bottleneck: In reasoning RL, rollout generation often takes over 3× longer than the backpropagation update. M1’s recurrent states eliminate large KV cache accesses, accelerating rollout generation and making RL on 24k–32k context windows practical.
- Throughput translates directly into accuracy: Because hybrid Mamba inference is memory-efficient, M1 generates substantially more tokens per second at large batch sizes. Under a fixed wall-clock time budget, M1 achieves higher accuracy via majority voting (Maj@k) than same-sized or smaller Transformers because it can sample far more paths in parallel.
Key Results
- Reasoning Parity: M1-3B performs on par with
DeepSeek-R1-Distill-Qwen-1.5Band competitive 7B models across standard math benchmarks: - AIME 2024: 28.9% Pass@1 (50.5% Maj@32)
- AIME 2025: 23.5% Pass@1 (34.6% Maj@32)
- MATH500: 82.1% Pass@1 (91.8% Maj@32)
- OlympiadBench: 47.3% Pass@1 (80.1% Maj@32)
- Decoding Latency and Throughput: Compared to
Llama-3.2-3Bserved on vLLM, M1 provides a 3× speedup at batch size 512 (decoding length 4,096) and remains 2.64× faster at long sequence lengths (~20k tokens). - Time-Constrained Scaling: M1 hits an optimal throughput of 15,169 tokens/sec on an H100 GPU compared to 7,263 tokens/sec for DeepSeek-R1-Distill-Qwen-1.5B, yielding higher accuracy per second of compute during self-consistency search.
Limitations and Future Work
- Software Optimizations: M1’s attention layers were not yet fully merged into vLLM’s custom kernels. Integrating optimized hybrid kernels (such as NVIDIA’s hybrid Mamba kernel) could yield even larger speedups.
- Teacher Model Dependency: The authors distilled from Llama-3.2 rather than Qwen2.5-Math-1.5B because the OpenMathInstruct dataset was closer to Llama's distribution (cross-entropy was 0.5 for Llama vs. >1.8 for Qwen). Creating Qwen-native distillation datasets remains future work.
- Scale: The study focuses on 3B models; validating whether these distillation and RL training dynamics scale effectively to 7B–70B parameter hybrid models remains an open exploration area.
Summary updated