MobileLLM-R1: Exploring the Limits of Sub-Billion Language Model Reasoners with Open Training Recipes

Changsheng Zhao, Ernie Chang, Zechun Liu, Chia-Jung Chang, Wei Wen, Chen Lai, Sheng Cao, Yuandong Tian, Raghuraman Krishnamoorthi, Yangyang Shi, Vikas Chandra

TL;DR

MobileLLM-R1 demonstrates that sub-billion parameter models can achieve state-of-the-art reasoning capabilities using only open-source data and a fraction of the tokens typically required by proprietary models. By employing principled data curation and dynamic mixing strategies, the authors developed a 950M parameter model that rivals or surpasses models like Qwen3-0.6B despite being trained on nearly 90% less data.

What Problem Does This Paper Address?

The AI community has long operated under two major assumptions: first, that complex reasoning (like chain-of-thought) only emerges in very large models, and second, that developing these capabilities requires training on massive corpora exceeding 10 trillion tokens.

While recent "small" models have challenged the first assumption, they often rely on massive, proprietary datasets. The authors of this paper tackle the second assumption, asking: What is the most effective recipe to endow small models with strong reasoning using limited, open-source data? They specifically focus on the "on-device" regime, where memory and compute constraints make efficiency paramount.

Methods

The researchers moved away from the "more is better" approach to data, instead focusing on capability-aware curation. Their pipeline consists of three main stages:

  1. Pre-training (4.2T tokens): They used a two-phase approach. Phase 1 focuses on broad web/educational coverage, while Phase 2 shifts toward specialized reasoning datasets. To determine the best data mix, they performed a Leave-One-Out (LOO) analysis, measuring how removing specific datasets impacted the model's Negative Log-Likelihood (NLL) across different domains.
  2. Mid-training (200B tokens): This stage focuses on "knowledge compression." They introduced a data-model co-evolution strategy where they calculated "influence scores" for training samples. Samples with negative or zero influence (meaning they no longer help the model learn) were discarded, iteratively refining the dataset as the model became more capable.
  3. Post-training: The models underwent instruction alignment (Tulu3-SFT) followed by reasoning-specific Supervised Fine-Tuning (SFT) using math, science, and code datasets to elicit long chain-of-thought traces.
Figure 2: Overall training pipeline of MobileLLM-R1.

Datasets

The study relies entirely on open-source data:

  • Pre-training: StarCoder, OpenWebMath, FineWeb-Edu, Wikipedia, Arxiv, StackExchange, and Algebraic Stack.
  • Mid-training: Dolmino (DCLM baseline), FLAN, and domain-specific subsets from Nemotron and StarCoder.
  • Post-training: Tulu3-SFT, OpenMathReasoning, OpenScienceReasoning-2, and OpenCodeReasoning-2.
  • Evaluation: A battery of reasoning benchmarks including HumanEval (code), GSM8K (math), MATH500, AIME'24, and LiveCodeBench (LCBv6).

Key Insights and Ideas

  • Web Data as "Glue": The LOO analysis revealed that FINEWEB-EDU was the most critical dataset. Removing it caused the largest degradation across all capabilities (math, code, and knowledge), suggesting large-scale web data acts as a foundational "glue" that binds heterogeneous domains together.
  • Unexpected Transfers: Interestingly, the authors found that STARCODER (code data) benefited math performance more than OPENWEBMATH (math data) benefited code performance.
  • The Convergence of Influence: During mid-training, influence scores for most samples eventually concentrated around zero. This indicates a convergence point where the model has effectively "exhausted" the informative content of the dataset, providing a principled signal for when to stop training or change data.
Figure 3: Leave-one-out analysis of pretraining data. The y-axis represents the normalized negative log-likelihood (NLL) on the curated capability probing datasets. We systematically exclude individual datasets—StarCoder, OpenWebMath, FineWeb-Edu, Wiki, Arxiv, StackExchange, and Algebraic Stack – to quantify their per-token contributions to downstream performance by comparing them with using full set of data. These trajectories reveal how beneficial each dataset is and how its impact evolves throughout training. Removing FineWeb-Edu yields the largest cross-domain degradation, likely attributable to its board web-based composition that connects diverse domain.

Key Results

MobileLLM-R1 sets a new Pareto frontier for accuracy vs. training efficiency.

  • Superior Reasoning: MobileLLM-R1-950M achieved an AIME score of 15.5, dwarfing other open models like OLMo-2-1.48B (0.6) and SmolLM-2-1.7B (0.3).
  • Data Efficiency: Despite using only 11.7% of the tokens compared to Qwen3's 36T-token corpus, the 950M model matched or surpassed Qwen3-0.6B across multiple benchmarks.
  • Coding Prowess: On HumanEval, the 950M model reached 46.3% accuracy, significantly higher than Qwen3-0.6B (30.5%).
  • On-Device Performance: The 140M parameter variant sustains over 100 tokens/second on a Samsung Galaxy S22, making it highly practical for real-time on-device reasoning.
Figure 1: Pretrained model accuracy vs. training efficiency trade-off.

Limitations and Future Work

  • Small Model Floor: The authors noted that models below 400M parameters still struggle to produce reliable outputs on highly complex coding benchmarks like LiveCodeBench.
  • RL vs. SFT: While Reinforcement Learning (RL) helped the base models, the authors found that for these tiny models, high-quality SFT distillation from larger teachers (like Llama 3) was often more effective and stable than direct RL exploration.
  • Knowledge Trade-offs: Introducing heavy math and code data sometimes led to a slight trade-off in factual knowledge retention (MMLU scores), a common challenge in capacity-constrained models.

Future work may explore how to better balance these trade-offs and whether even more aggressive data distillation can push sub-100M models into the reasoning regime.

Summary updated

Abstract

The paradigm shift in large language models (LLMs) from instinctive responses to chain-of-thought (CoT) reasoning has fueled two prevailing assumptions: (1) reasoning capabilities only emerge in sufficiently large models, and (2) such capabilities require training on massive datasets. While the first assumption has already been challenged by recent sub-billion-parameter reasoning models such as Qwen3-0.6B and DeepSeek distilled variants, the second remains largely unquestioned. In this work, we revisit the necessity of scaling to extremely large corpora (>10T tokens) for reasoning emergence. By carefully curating and resampling open-source datasets that we identify as beneficial under our designed metrics, we demonstrate that strong reasoning abilities can emerge with far less data. Specifically, we show that only ~2T tokens of high-quality data are sufficient, and pre-training with 4.2T tokens on the dataset resampled from these ~2T tokens, followed by a established post-training procedure, enables the development of MobileLLM-R1, a series of sub-billion-parameter reasoning models that substantially outperform prior models trained on fully open-sourced data. For example, MobileLLM-R1-950M achieves an AIME score of 15.5, compared to just 0.6 for OLMo-2-1.48B and 0.3 for SmolLM-2-1.7B. Remarkably, despite being trained on only 11.7% of the tokens compared to Qwen3's proprietary 36T-token corpus for pretraining, MobileLLM-R1-950M matches or surpasses Qwen3-0.6B across multiple reasoning benchmarks. To facilitate further research in this direction, we have made the models (https://huggingface.co/collections/facebook/mobilellm-r1) and code (https://github.com/facebookresearch/MobileLLM-R1) publicly available, along with the complete training recipe, data sources, and data mixing ratios.
BibTeX
@inproceedings{zhao2026mobilellm,
  author = {Changsheng Zhao and Ernie Chang and Zechun Liu and Chia-Jung Chang and Wei Wen and Chen Lai and Sheng Cao and Yuandong Tian and Raghuraman Krishnamoorthi and Yangyang Shi and Vikas Chandra},
  title = {{MobileLLM-R1: Exploring the Limits of Sub-Billion Language Model Reasoners with Open Training Recipes}},
  booktitle = {International Conference on Learning Representations},
  year = {2026},
  url = {https://openreview.net/forum?id=GMlZt4fZSY},
}
Figure