Understanding the Role of Training Data in Test-Time Scaling

Adel Javanmard, Baharan Mirzasoleiman, Vahab Mirrokni

Recent breakthroughs in AI, such as OpenAI’s o1 and DeepSeek R1, have demonstrated a fascinating phenomenon: Large Language Models (LLMs) become significantly more capable when allowed to "think" longer at test time using Chain-of-Thought (CoT) reasoning. By breaking down complex problems into additional steps, backtracking, and correcting mistakes, these models achieve remarkable performance leaps.

However, while the empirical results are stunning, the underlying mechanics remain largely a black box. What specific properties of training data enable a model to benefit from extra test-time compute? Can test-time compute act as a direct substitute for training-time compute, such as shorter context windows? And perhaps most intriguingly, why do models sometimes fall into the trap of "overthinking," performing worse when given more steps to reason?

A recent paper, Understanding the Role of Training Data in Test-Time Scaling, provides a rigorous theoretical foundation to answer these questions. By studying Transformers trained for in-context weight prediction—a form of linear regression—the authors uncover the hidden dynamics of test-time scaling.

Modeling Chain-of-Thought as Optimization

To peer inside the black box, the researchers modeled the Transformer’s CoT process as an iterative optimization algorithm. They mathematically proved that a trained Linear Self-Attention (LSA) model effectively implements a multi-step (pseudo-) Newton’s method during test time.

A key technical contribution of this work is the formal definition of Task Hardness. The authors define "hardness" based on the feature covariance matrix ($\Lambda$) of a task, specifically as $Hard(\Lambda) = \text{trace}(\Lambda) / \lambda_{\min}(\Lambda)$. In simpler terms, a "hard" task requires the model to learn many different "skills" (represented by the eigenvectors of the covariance matrix), especially those subtle skills with small eigenvalues that are easily missed during training.

Key Insights: Trade-offs, Overthinking, and Task Selection

Through this theoretical lens, the paper reveals three critical insights about how training data dictates test-time performance.

1. The Compute Trade-off

There is a direct, quantifiable exchange rate between training and testing compute. The authors derived theoretical bounds showing that test error decreases as a function of both training context length ($n$) and test-time CoT steps ($k$).

Specifically, the error follows a scaling law roughly proportional to $(1 + \frac{n}{1 + Hard(\Lambda)})^{-2k}$. This means that if you increase test-time compute (more CoT steps, $k$), you can achieve the same accuracy even if the model was trained on significantly shorter context windows ($n$).

Estimation error vs. computational complexity, illustrating the trade-off between training context length ($n$), task hardness ($H$), and test-time compute.

As task hardness increases, a model requires either more training data or more thinking steps to compensate and drive down the estimation error.

2. The "Overthinking" Trap

Perhaps the most striking insight is the phenomenon of overthinking. If the specific "skills" (directions in the covariance matrix) required to solve a downstream test task were not sufficiently covered in the training data, increasing test-time compute actually amplifies the model's errors.

Instead of reasoning its way to a correct answer, the model essentially "hallucinates" a solution based on its biased, incomplete training. In these mismatched scenarios, the longer the model thinks, the worse its performance becomes.

3. Hard Tasks are Better Teachers

To make a model truly "scaling-ready," training on easy, perfectly balanced data is insufficient. The authors formulated an optimization problem to determine the best mixture of training tasks, concluding that models should be trained on a diverse set of "hard" tasks—those with long-tailed eigenvalue spectra. This forces the model to learn a broader, more robust range of skills that it can successfully deploy during extended test-time reasoning.

Empirical Validation

The researchers validated their theoretical claims across a spectrum of models and datasets:

  • Synthetic & GPT-2 Experiments: Using synthetic linear regression and GPT-2 architectures, they confirmed that for a fixed error rate, increasing CoT steps allowed for a deliberate reduction in training context length.
  • Real-World Reasoning (Qwen 2.5): Testing on the OMEGA dataset (a real-world reasoning benchmark), the authors fine-tuned Qwen 2.5-7B-Instruct. When fine-tuned on the Greatest Common Divisor (GCD) task, Qwen-GCD saw its accuracy jump from 30.39% to 75% with short CoT. Conversely, Qwen-Poly (fine-tuned on a mismatched polynomial root task) exhibited the exact "overthinking" effect predicted by the theory: when tested on GCD, its performance dropped from 29% to 20.83% as CoT length increased.

Looking Ahead

While this paper provides a vital mathematical bridge for understanding test-time scaling, its primary limitation is that the core proofs rely on linear regression tasks and linear self-attention.

Future work will need to extend these theories to non-linear data generation and explore the impact of non-linear activation functions (like ReLU or GeLU) on CoT dynamics. Furthermore, understanding how reinforcement learning (RL) interacts with these data properties could unlock even more efficient and powerful test-time scaling paradigms. For now, this research makes one thing abundantly clear: the quality and hardness of your training data fundamentally dictate the ceiling of your model's ability to reason.

Summary updated

Abstract

Test-time scaling improves the reasoning capabilities of large language models (LLMs) by allocating extra compute to generate longer Chains-of-Thoughts (CoTs). This enables models to tackle more complex problem by breaking them down into additional steps, backtracking, and correcting mistakes. Despite its strong performance--demonstrated by OpenAI's o1 and DeepSeek R1, the conditions in the training data under which long CoTs emerge, and when such long CoTs improve the performance, remain unclear. In this paper, we study the performance of test-time scaling for transformers trained on an in-context weight prediction task for linear regression. Our analysis provides a theoretical explanation for several intriguing observations: First, at any fixed test error, increasing test-time compute allows us to reduce the number of in-context examples (context length) in training prompts. Second, if the skills required to solve a downstream task are not sufficiently present in the training data, increasing test-time compute can harm performance. Finally, we characterize task hardness via the smallest eigenvalue of its feature covariance matrix and show that training on a diverse, relevant, and hard set of tasks results in best performance for test-time scaling. We confirm our findings with experiments on large, nonlinear transformer architectures.
BibTeX
@inproceedings{javanmard2026understanding,
  author = {Adel Javanmard and Baharan Mirzasoleiman and Vahab Mirrokni},
  title = {{Understanding the Role of Training Data in Test-Time Scaling}},
  booktitle = {International Conference on Learning Representations},
  year = {2026},
  url = {https://openreview.net/forum?id=Y9FfDNa2nJ},
}
Figure