Thinking Augmented Pre-training

Liang Wang, Nan Yang, Shaohan Huang, Li Dong, Furu Wei

Abstract

This paper introduces a simple and scalable approach to improve the data efficiency of large language model (LLM) training by augmenting existing text data with thinking trajectories. The compute for pre-training LLMs has been growing at an unprecedented rate, while the availability of high-quality data remains limited. Consequently, maximizing the utility of available data constitutes a significant research challenge. A primary impediment is that certain high-quality tokens are difficult to learn given a fixed model capacity, as the underlying rationale for a single token can be exceptionally complex and deep. To address this issue, we propose Thinking augmented Pre-Training (TPT), a universal methodology that augments text with automatically generated thinking trajectories. Such augmentation effectively increases the volume of the training data and makes high-quality tokens more learnable through step-by-step reasoning and decomposition. We apply TPT across diverse training configurations up to 100B tokens, encompassing pre-training with both constrained and abundant data, as well as mid-training from strong open-source checkpoints. Experimental results indicate that our method substantially improves the performance of LLMs across various model sizes and families. Notably, TPT enhances the data efficiency of LLM pre-training by a factor of 3. For a 3B parameter model, it improves the post-training performance by over 10% on several challenging reasoning benchmarks.

Introduction

As Large Language Model (LLM) development approaches a "data wall," the supply of high-quality, human-authored text is becoming increasingly finite. Standard next-token prediction often struggles with complex concepts because the underlying rationale for a specific token is missing from the training corpus. Without intermediate steps, a model must perform an exceptionally deep cognitive leap to predict the correct output. Thinking Augmented Pre-training (TPT) addresses this by bridging the gap between raw data and the reasoning required to generate it.

Methodology: Thinking Augmented Pre-training (TPT)

TPT is a data engineering strategy that transforms standard pre-training corpora into a more "LLM-friendly" format before training begins. Unlike Reinforcement Learning approaches that require expensive online rollouts, TPT is a scalable offline pipeline.

Comparison of Vanilla vs. Thinking Augmented data formats.

The process follows three primary steps:

  1. Thinking Generation: A document from a standard corpus (e.g., FineWeb-Edu) is processed by an off-the-shelf reasoning model. This "teacher" model generates a "thinking trajectory"—a step-by-step analysis of the text.
  2. Data Augmentation: The original document is concatenated with the generated thinking trajectory to create an augmented training sample.
  3. Standard Training: The target model is trained using standard next-token prediction loss on this enriched data.

This method naturally implements dynamic compute allocation. Difficult domains like mathematics or physics trigger longer reasoning trajectories from the teacher model, effectively up-sampling the training budget for high-value, complex concepts.

Experimental Results

Evaluation was conducted across a wide range of benchmarks, including GSM8k and MATH for reasoning, MMLU for general knowledge, and HumanEval for coding.

Performance comparison showing 3x data efficiency and parity with LLaMA-3.1-8B.

Data Efficiency and Scaling

The core finding is a 3x improvement in data efficiency. As illustrated in the performance curves, TPT allows a model to reach the same performance level as a vanilla model using only one-third of the tokens. Remarkably, after training on just 100 billion tokens, the TPT-8B model achieved performance comparable to LLaMA-3.1-8B, which was trained on 15 trillion tokens—a 150x difference in data volume.

Reasoning Performance

TPT yields significant jumps in reasoning capabilities. On the GSM8k benchmark, TPT-8B improved accuracy from 19.2% (vanilla) to 50.1%. On the MATH benchmark, it more than doubled the baseline score, rising from 9.1% to 21.8%. These improvements suggest that by seeing the "why" behind a token, the model learns to generalize reasoning patterns rather than simply memorizing statistical associations.

Conclusion

Thinking Augmented Pre-training demonstrates that the utility of existing data can be significantly amplified by making the underlying reasoning explicit. By achieving frontier-level performance with a fraction of the traditional data requirements, TPT provides a viable path forward in an era of data scarcity. Future research will focus on scaling these gains to multi-trillion token regimes and further optimizing the teacher-student learning curriculum.

Summary updated

Abstract

This paper introduces a simple and scalable approach to improve the data efficiency of large language model (LLM) training by augmenting existing text data with thinking trajectories. The compute for pre-training LLMs has been growing at an unprecedented rate, while the availability of high-quality data remains limited. Consequently, maximizing the utility of available data constitutes a significant research challenge. A primary impediment is that certain high-quality tokens are difficult to learn given a fixed model capacity, as the underlying rationale for a single token can be exceptionally complex and deep. To address this issue, we propose Thinking augmented Pre-Training (TPT), a universal methodology that augments text with automatically generated thinking trajectories. Such augmentation effectively increases the volume of the training data and makes high-quality tokens more learnable through step-by-step reasoning and decomposition. We apply TPT across diverse training configurations up to $100$B tokens, encompassing pre-training with both constrained and abundant data, as well as mid-training from strong open-source checkpoints. Experimental results indicate that our method substantially improves the performance of LLMs across various model sizes and families. Notably, TPT enhances the data efficiency of LLM pre-training by a factor of $3$. For a $3$B parameter model, it improves the post-training performance by over $10\%$ on several challenging reasoning benchmarks.
BibTeX
@misc{wang2025thinking,
  author = {Liang Wang and Nan Yang and Shaohan Huang and Li Dong and Furu Wei},
  title = {{Thinking Augmented Pre-training}},
  year = {2025},
  eprint = {2509.20186},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.20186},
}
Figure