TL;DR
MobileLLM-R1 demonstrates that sub-billion parameter models can achieve state-of-the-art reasoning capabilities using only open-source data and a fraction of the tokens typically required by proprietary models. By employing principled data curation and dynamic mixing strategies, the authors developed a 950M parameter model that rivals or surpasses models like Qwen3-0.6B despite being trained on nearly 90% less data.
What Problem Does This Paper Address?
The AI community has long operated under two major assumptions: first, that complex reasoning (like chain-of-thought) only emerges in very large models, and second, that developing these capabilities requires training on massive corpora exceeding 10 trillion tokens.
While recent "small" models have challenged the first assumption, they often rely on massive, proprietary datasets. The authors of this paper tackle the second assumption, asking: What is the most effective recipe to endow small models with strong reasoning using limited, open-source data? They specifically focus on the "on-device" regime, where memory and compute constraints make efficiency paramount.
Methods
The researchers moved away from the "more is better" approach to data, instead focusing on capability-aware curation. Their pipeline consists of three main stages:
- Pre-training (4.2T tokens): They used a two-phase approach. Phase 1 focuses on broad web/educational coverage, while Phase 2 shifts toward specialized reasoning datasets. To determine the best data mix, they performed a Leave-One-Out (LOO) analysis, measuring how removing specific datasets impacted the model's Negative Log-Likelihood (NLL) across different domains.
- Mid-training (200B tokens): This stage focuses on "knowledge compression." They introduced a data-model co-evolution strategy where they calculated "influence scores" for training samples. Samples with negative or zero influence (meaning they no longer help the model learn) were discarded, iteratively refining the dataset as the model became more capable.
- Post-training: The models underwent instruction alignment (Tulu3-SFT) followed by reasoning-specific Supervised Fine-Tuning (SFT) using math, science, and code datasets to elicit long chain-of-thought traces.
Datasets
The study relies entirely on open-source data:
- Pre-training: StarCoder, OpenWebMath, FineWeb-Edu, Wikipedia, Arxiv, StackExchange, and Algebraic Stack.
- Mid-training: Dolmino (DCLM baseline), FLAN, and domain-specific subsets from Nemotron and StarCoder.
- Post-training: Tulu3-SFT, OpenMathReasoning, OpenScienceReasoning-2, and OpenCodeReasoning-2.
- Evaluation: A battery of reasoning benchmarks including HumanEval (code), GSM8K (math), MATH500, AIME'24, and LiveCodeBench (LCBv6).
Key Insights and Ideas
- Web Data as "Glue": The LOO analysis revealed that
FINEWEB-EDUwas the most critical dataset. Removing it caused the largest degradation across all capabilities (math, code, and knowledge), suggesting large-scale web data acts as a foundational "glue" that binds heterogeneous domains together. - Unexpected Transfers: Interestingly, the authors found that
STARCODER(code data) benefited math performance more thanOPENWEBMATH(math data) benefited code performance. - The Convergence of Influence: During mid-training, influence scores for most samples eventually concentrated around zero. This indicates a convergence point where the model has effectively "exhausted" the informative content of the dataset, providing a principled signal for when to stop training or change data.
Key Results
MobileLLM-R1 sets a new Pareto frontier for accuracy vs. training efficiency.
- Superior Reasoning: MobileLLM-R1-950M achieved an AIME score of 15.5, dwarfing other open models like OLMo-2-1.48B (0.6) and SmolLM-2-1.7B (0.3).
- Data Efficiency: Despite using only 11.7% of the tokens compared to Qwen3's 36T-token corpus, the 950M model matched or surpassed Qwen3-0.6B across multiple benchmarks.
- Coding Prowess: On HumanEval, the 950M model reached 46.3% accuracy, significantly higher than Qwen3-0.6B (30.5%).
- On-Device Performance: The 140M parameter variant sustains over 100 tokens/second on a Samsung Galaxy S22, making it highly practical for real-time on-device reasoning.
Limitations and Future Work
- Small Model Floor: The authors noted that models below 400M parameters still struggle to produce reliable outputs on highly complex coding benchmarks like LiveCodeBench.
- RL vs. SFT: While Reinforcement Learning (RL) helped the base models, the authors found that for these tiny models, high-quality SFT distillation from larger teachers (like Llama 3) was often more effective and stable than direct RL exploration.
- Knowledge Trade-offs: Introducing heavy math and code data sometimes led to a slight trade-off in factual knowledge retention (MMLU scores), a common challenge in capacity-constrained models.
Future work may explore how to better balance these trade-offs and whether even more aggressive data distillation can push sub-100M models into the reasoning regime.
Summary updated