Thinking Augmented Pre-training

Liang Wang, Nan Yang, Shaohan Huang, Li Dong, Furu WeiSubmission to ICLR 2026 · Rejected

TL;DR

Thinking Augmented Pre-Training (TPT) enhances the data efficiency of large language model (LLM) training by augmenting pre-training documents with synthetic "thinking trajectories" generated by off-the-shelf reasoning models. This simple document-level augmentation improves pre-training data efficiency by 3× and dramatically boosts post-training reasoning performance across model sizes from 1.5B to 8B parameters.

What Problem Does This Paper Address?

Pre-training compute continues to grow rapidly, but high-quality human-authored web data is finite and largely exhausted by frontier models. A major obstacle in extracting utility from existing corpora is that some valuable, high-quality tokens—such as the final answer to an intricate math problem—are extremely difficult for an LLM to predict directly via standard next-token prediction. Because the underlying rationale involves complex multi-step reasoning, models often resort to poor memorization rather than learning generalizable principles.

Methods

The authors propose Thinking augmented Pre-Training (TPT), which transforms raw pre-training text into an LLM-friendly format:

  • Trajectory Generation: Given a document $d$, an off-the-shelf model (e.g., DeepSeek-R1-Distill-Qwen-7B or Qwen3-8B) generates a synthetic "expert thought process" $t$ analyzing the text using a prompt that instructs the model to explain complex details via the Feynman technique. Generation stops at the </think> tag.
  • Data Concatenation: The raw document and thinking trajectory are concatenated: $x = [d; t]$.
  • Standard Next-Token Prediction: The model is trained using the standard autoregressive language modeling objective over the concatenated text $x$: $$\min \mathcal{L} = -\frac{1}{N} \sum_{i=1}^N \log p(x_i \mid x_{<i})$$
  • Unlike reinforcement learning pre-training (RPT) or latent thought methods, TPT requires no online rollouts or complex token-level RL algorithms—it operates purely as a scalable offline data engineering pipeline.

Datasets

  • Pre-training / Mid-training Corpora:
  • FineWeb-Edu: High-quality educational subset of Common Crawl.
  • MegaMath-Web-Pro-Max: Filtered math-intensive web text (~70B tokens). FineWeb-Edu was mixed at a 0.125 weight.
  • Supervised Fine-Tuning (SFT):
  • Mixture-of-Thoughts: A public dataset of 350k reasoning examples distilled from DeepSeek-R1 (~2B tokens).
  • Evaluation Benchmarks:
  • Base Models: GSM8k, MATH, BoolQ, MMLU, MMLU-Pro.
  • Post-SFT Models: MATH-500, AIME 2024, AIME 2025, GSM8k, HMMT 2025, HumanEval, LiveCodeBench (LCB v4/v5), GPQA-Diamond, MMLU-Pro, and JEEBench.

Key Insights and Ideas

  • Training-Time Compute Scaling via Reasoning Length: Just as inference-time compute scaling (e.g., in OpenAI o1 or DeepSeek-R1) expends more tokens on hard queries, TPT acts as an implicit, natural data up-sampler. Analysis shows that trajectories are ~50% longer for texts with "Advanced Reasoning" compared to "No Reasoning", and significantly longer for STEM topics like Mathematics and Physics. Difficult content automatically gets more gradient updates.
  • Surpassing the Teacher: Models trained on TPT data can outperform the generator model itself (e.g., an 8B model trained with TPT outperforms LLaMA-3.1-8B-Instruct and rivals distilled 7B models), proving that decomposing text into explainable steps enables learners to internalize knowledge beyond superficial imitation.
  • Smaller Generator Models Can Be Better: In ablation experiments, generating thinking trajectories with a smaller 1.5B distillation model (DeepSeek-R1-Distill-Qwen-1.5B) outperformed using a 7B model. Smaller models may produce more accessible, didactic step-by-step reasoning better suited for downstream learning.
  • Overcoming Data Exhaustion: In data-constrained experiments (capping raw unique tokens at 10B and training for 40B tokens), vanilla pre-training plateaus after 1 epoch, while TPT continues to scale smoothly without overfitting.

Key Results

  • Data Efficiency (Pre-training from scratch): Under an abundant data regime (100B tokens), an 8B model trained with TPT achieves benchmark performance comparable to LLaMA-3.1-8B (which was trained on 15T tokens, 150× more data), cutting token requirements by a factor of 3× relative to vanilla pre-training.
  • Post-SFT Reasoning Uplift: After SFT on the 2B-token Mixture-of-Thoughts dataset:
  • TPT-8B achieved 35.2% on AIME24 and 82.4% on MATH-500, drastically outperforming both the vanilla 8B baseline (1.0% and 33.8%) and LLaMA-3.1-8B-Instruct (5.4% and 49.4%).
  • Mid-Training Across Model Families:
  • On LLaMA-3.2-3B, 100B tokens of TPT mid-training improved AIME24 from 5.8% to 18.6% (a >3× jump) and LiveCodeBench from 13.9% to 20.0% compared to direct SFT.
  • On Qwen2.5-7B, TPT mid-training boosted MATH-500 to 92.5% and AIME24 to 57.5% (up from 50.5% for OpenR1-Qwen2.5-7B).

Limitations and Future Work

  • Compute Overhead of Generation: Generating trajectories for 100B training tokens required approximately 20,000 A100 GPU hours, which adds an upfront inference cost prior to pre-training.
  • Prompt Sensitivity: The current pipeline relies on a single fixed prompt template; automatic prompt optimization or domain-specific prompting strategies remain unexplored.
  • Scale: Experiments were capped at 100B tokens and 8B parameters; validating TPT on trillion-token pre-training regimes and 70B+ scale models remains future work.

Summary updated

Abstract

This paper introduces a simple and scalable approach to improve the data efficiency of large language model (LLM) training by augmenting existing text data with thinking trajectories. The compute for pre-training LLMs has been growing at an unprecedented rate, while the availability of high-quality data remains limited. Consequently, maximizing the utility of available data constitutes a significant research challenge. A primary impediment is that certain high-quality tokens are difficult to learn given a fixed model capacity, as the underlying rationale for a single token can be exceptionally complex and deep. To address this issue, we propose Thinking augmented Pre-Training (TPT), a universal methodology that augments text with automatically generated thinking trajectories. Such augmentation effectively increases the volume of the training data and makes high-quality tokens more learnable through step-by-step reasoning and decomposition. We apply TPT across diverse training configurations up to $100$B tokens, encompassing pre-training with both constrained and abundant data, as well as mid-training from strong open-source checkpoints. Experimental results indicate that our method substantially improves the performance of LLMs across various model sizes and families. Notably, TPT enhances the data efficiency of LLM pre-training by a factor of $3$. For a $3$B parameter model, it improves the post-training performance by over $10%$ on several challenging reasoning benchmarks.
BibTeX
@misc{wang2025thinking,
  author = {Liang Wang and Nan Yang and Shaohan Huang and Li Dong and Furu Wei},
  title = {{Thinking Augmented Pre-training}},
  year = {2025},
  eprint = {2509.20186},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.20186},
}

Loading related content...

Figure