Abstract
This paper introduces a simple and scalable approach to improve the data efficiency of large language model (LLM) training by augmenting existing text data with thinking trajectories. The compute for pre-training LLMs has been growing at an unprecedented rate, while the availability of high-quality data remains limited. Consequently, maximizing the utility of available data constitutes a significant research challenge. A primary impediment is that certain high-quality tokens are difficult to learn given a fixed model capacity, as the underlying rationale for a single token can be exceptionally complex and deep. To address this issue, we propose Thinking augmented Pre-Training (TPT), a universal methodology that augments text with automatically generated thinking trajectories. Such augmentation effectively increases the volume of the training data and makes high-quality tokens more learnable through step-by-step reasoning and decomposition. We apply TPT across diverse training configurations up to 100B tokens, encompassing pre-training with both constrained and abundant data, as well as mid-training from strong open-source checkpoints. Experimental results indicate that our method substantially improves the performance of LLMs across various model sizes and families. Notably, TPT enhances the data efficiency of LLM pre-training by a factor of 3. For a 3B parameter model, it improves the post-training performance by over 10% on several challenging reasoning benchmarks.
Introduction
As Large Language Model (LLM) development approaches a "data wall," the supply of high-quality, human-authored text is becoming increasingly finite. Standard next-token prediction often struggles with complex concepts because the underlying rationale for a specific token is missing from the training corpus. Without intermediate steps, a model must perform an exceptionally deep cognitive leap to predict the correct output. Thinking Augmented Pre-training (TPT) addresses this by bridging the gap between raw data and the reasoning required to generate it.
Methodology: Thinking Augmented Pre-training (TPT)
TPT is a data engineering strategy that transforms standard pre-training corpora into a more "LLM-friendly" format before training begins. Unlike Reinforcement Learning approaches that require expensive online rollouts, TPT is a scalable offline pipeline.
The process follows three primary steps:
- Thinking Generation: A document from a standard corpus (e.g., FineWeb-Edu) is processed by an off-the-shelf reasoning model. This "teacher" model generates a "thinking trajectory"—a step-by-step analysis of the text.
- Data Augmentation: The original document is concatenated with the generated thinking trajectory to create an augmented training sample.
- Standard Training: The target model is trained using standard next-token prediction loss on this enriched data.
This method naturally implements dynamic compute allocation. Difficult domains like mathematics or physics trigger longer reasoning trajectories from the teacher model, effectively up-sampling the training budget for high-value, complex concepts.
Experimental Results
Evaluation was conducted across a wide range of benchmarks, including GSM8k and MATH for reasoning, MMLU for general knowledge, and HumanEval for coding.
Data Efficiency and Scaling
The core finding is a 3x improvement in data efficiency. As illustrated in the performance curves, TPT allows a model to reach the same performance level as a vanilla model using only one-third of the tokens. Remarkably, after training on just 100 billion tokens, the TPT-8B model achieved performance comparable to LLaMA-3.1-8B, which was trained on 15 trillion tokens—a 150x difference in data volume.
Reasoning Performance
TPT yields significant jumps in reasoning capabilities. On the GSM8k benchmark, TPT-8B improved accuracy from 19.2% (vanilla) to 50.1%. On the MATH benchmark, it more than doubled the baseline score, rising from 9.1% to 21.8%. These improvements suggest that by seeing the "why" behind a token, the model learns to generalize reasoning patterns rather than simply memorizing statistical associations.
Conclusion
Thinking Augmented Pre-training demonstrates that the utility of existing data can be significantly amplified by making the underlying reasoning explicit. By achieving frontier-level performance with a fraction of the traditional data requirements, TPT provides a viable path forward in an era of data scarcity. Future research will focus on scaling these gains to multi-trillion token regimes and further optimizing the teacher-student learning curriculum.
Summary updated