TL;DR
Thinking Augmented Pre-Training (TPT) enhances the data efficiency of large language model (LLM) training by augmenting pre-training documents with synthetic "thinking trajectories" generated by off-the-shelf reasoning models. This simple document-level augmentation improves pre-training data efficiency by 3× and dramatically boosts post-training reasoning performance across model sizes from 1.5B to 8B parameters.
What Problem Does This Paper Address?
Pre-training compute continues to grow rapidly, but high-quality human-authored web data is finite and largely exhausted by frontier models. A major obstacle in extracting utility from existing corpora is that some valuable, high-quality tokens—such as the final answer to an intricate math problem—are extremely difficult for an LLM to predict directly via standard next-token prediction. Because the underlying rationale involves complex multi-step reasoning, models often resort to poor memorization rather than learning generalizable principles.
Methods
The authors propose Thinking augmented Pre-Training (TPT), which transforms raw pre-training text into an LLM-friendly format:
- Trajectory Generation: Given a document $d$, an off-the-shelf model (e.g., DeepSeek-R1-Distill-Qwen-7B or Qwen3-8B) generates a synthetic "expert thought process" $t$ analyzing the text using a prompt that instructs the model to explain complex details via the Feynman technique. Generation stops at the
</think>tag. - Data Concatenation: The raw document and thinking trajectory are concatenated: $x = [d; t]$.
- Standard Next-Token Prediction: The model is trained using the standard autoregressive language modeling objective over the concatenated text $x$: $$\min \mathcal{L} = -\frac{1}{N} \sum_{i=1}^N \log p(x_i \mid x_{<i})$$
- Unlike reinforcement learning pre-training (RPT) or latent thought methods, TPT requires no online rollouts or complex token-level RL algorithms—it operates purely as a scalable offline data engineering pipeline.
Datasets
- Pre-training / Mid-training Corpora:
- FineWeb-Edu: High-quality educational subset of Common Crawl.
- MegaMath-Web-Pro-Max: Filtered math-intensive web text (~70B tokens). FineWeb-Edu was mixed at a 0.125 weight.
- Supervised Fine-Tuning (SFT):
- Mixture-of-Thoughts: A public dataset of 350k reasoning examples distilled from DeepSeek-R1 (~2B tokens).
- Evaluation Benchmarks:
- Base Models: GSM8k, MATH, BoolQ, MMLU, MMLU-Pro.
- Post-SFT Models: MATH-500, AIME 2024, AIME 2025, GSM8k, HMMT 2025, HumanEval, LiveCodeBench (LCB v4/v5), GPQA-Diamond, MMLU-Pro, and JEEBench.
Key Insights and Ideas
- Training-Time Compute Scaling via Reasoning Length: Just as inference-time compute scaling (e.g., in OpenAI o1 or DeepSeek-R1) expends more tokens on hard queries, TPT acts as an implicit, natural data up-sampler. Analysis shows that trajectories are ~50% longer for texts with "Advanced Reasoning" compared to "No Reasoning", and significantly longer for STEM topics like Mathematics and Physics. Difficult content automatically gets more gradient updates.
- Surpassing the Teacher: Models trained on TPT data can outperform the generator model itself (e.g., an 8B model trained with TPT outperforms LLaMA-3.1-8B-Instruct and rivals distilled 7B models), proving that decomposing text into explainable steps enables learners to internalize knowledge beyond superficial imitation.
- Smaller Generator Models Can Be Better: In ablation experiments, generating thinking trajectories with a smaller 1.5B distillation model (DeepSeek-R1-Distill-Qwen-1.5B) outperformed using a 7B model. Smaller models may produce more accessible, didactic step-by-step reasoning better suited for downstream learning.
- Overcoming Data Exhaustion: In data-constrained experiments (capping raw unique tokens at 10B and training for 40B tokens), vanilla pre-training plateaus after 1 epoch, while TPT continues to scale smoothly without overfitting.
Key Results
- Data Efficiency (Pre-training from scratch): Under an abundant data regime (100B tokens), an 8B model trained with TPT achieves benchmark performance comparable to LLaMA-3.1-8B (which was trained on 15T tokens, 150× more data), cutting token requirements by a factor of 3× relative to vanilla pre-training.
- Post-SFT Reasoning Uplift: After SFT on the 2B-token Mixture-of-Thoughts dataset:
- TPT-8B achieved 35.2% on AIME24 and 82.4% on MATH-500, drastically outperforming both the vanilla 8B baseline (1.0% and 33.8%) and LLaMA-3.1-8B-Instruct (5.4% and 49.4%).
- Mid-Training Across Model Families:
- On LLaMA-3.2-3B, 100B tokens of TPT mid-training improved AIME24 from 5.8% to 18.6% (a >3× jump) and LiveCodeBench from 13.9% to 20.0% compared to direct SFT.
- On Qwen2.5-7B, TPT mid-training boosted MATH-500 to 92.5% and AIME24 to 57.5% (up from 50.5% for OpenR1-Qwen2.5-7B).
Limitations and Future Work
- Compute Overhead of Generation: Generating trajectories for 100B training tokens required approximately 20,000 A100 GPU hours, which adds an upfront inference cost prior to pre-training.
- Prompt Sensitivity: The current pipeline relies on a single fixed prompt template; automatic prompt optimization or domain-specific prompting strategies remain unexplored.
- Scale: Experiments were capped at 100B tokens and 8B parameters; validating TPT on trillion-token pre-training regimes and 70B+ scale models remains future work.
Summary updated