Predicting thinking time in Reasoning models

Hans Peter Lynsgøe Raaschou-jensen, Constanza Fierro, Anders SøgaardarXiv 2025 · DOI · arXiv

Abstract

Reasoning Language Models (RLMs) utilize test-time scaling to generate long internal chains of thought, a process that remains opaque to end-users. This research introduces a framework for real-time progress prediction within these models. By integrating linear probing, supervised fine-tuning (SFT), and reinforcement learning (RL), the authors demonstrate that a 1.5B parameter model can be trained to communicate its internal state via a progress bar. The resulting system achieves a 10% mean absolute error (MAE) for reasoning sequences up to 16,000 tokens, providing a significant step toward transparency in complex AI reasoning tasks.

Introduction

Modern Reasoning Language Models, such as DeepSeek-R1 and OpenAI’s o1, have shifted the paradigm of AI interaction by spending significant compute time on internal "chains of thought" before delivering a final response. While this "test-time scaling" improves accuracy, it creates a "black box" experience for users who cannot distinguish between a model that is nearly finished and one that has just begun a lengthy calculation.

Conceptual overview of a reasoning model providing real-time progress updates during its internal chain-of-thought.

This research addresses the lack of transparency by investigating whether RLMs possess an intrinsic awareness of their progress and if they can be trained to report it reliably to the user.

Methodology

The researchers employed a three-stage approach to develop and validate progress-aware models:

  1. Internal Probing: To determine if progress information is already present, linear classifiers were trained on the model's hidden states. These probes successfully predicted the current "bin" of progress (e.g., 20-30%), suggesting that models naturally encode their position within a task.
  2. Supervised Fine-Tuning (SFT): A 1.5B parameter model was fine-tuned on mathematical reasoning datasets augmented with progress markers (e.g., <progressbar>35%</progressbar>). To ensure the model relied on context rather than simple token counting, "Noisy Labels" (random jitter) and masking of previous markers were introduced during training.
  3. Reinforcement Learning (RL): Using Group Relative Policy Optimization (GRPO), the model was further refined. A tripartite reward system balanced prediction accuracy, formatting consistency (regular intervals), and the preservation of mathematical correctness.

Key Results

The study yielded several significant findings regarding the feasibility of real-time monitoring:

  • Predictive Accuracy: The optimized model achieved a 10% Mean Absolute Error (MAE) for sequences up to 16,000 tokens. As shown in Figure 3, the combination of SFT with noisy labels and RL provided the most stable performance across varying sequence lengths.
  • Stability and Format: While SFT-only models occasionally produced invalid predictions (e.g., progress exceeding 100%), RL-tuned models remained highly stable and adhered strictly to the required formatting.
  • The Backtracking Phenomenon: Interestingly, models occasionally decreased their progress estimates. This "backtracking" occurred when the model identified a mistake in its reasoning and effectively "restarted" its internal calculation, demonstrating a genuine reflection of its cognitive state.
Mean Absolute Error (MAE) of progress predictions across different training regimes and sequence lengths.

Discussion and Limitations

While the results are promising, the research highlights a "progress-performance trade-off." There is a minor decline in mathematical accuracy on the most difficult problems, suggesting that monitoring progress consumes a portion of the model's reasoning capacity. Furthermore, accuracy declines significantly on out-of-distribution sequences exceeding 32,000 tokens, indicating that the model's sense of progress is tied to the lengths encountered during training.

Future work will explore whether these techniques scale to larger models (70B+) and if progress awareness can be generalized to non-mathematical domains like creative writing or open-ended coding, where "the end" is less structurally defined.

Conclusion

By enabling reasoning models to report their own progress, we can transform the "black box" of test-time scaling into a transparent, manageable process. This work provides a foundation for better user interfaces and improved oversight of long-form AI reasoning.

Summary updated

Abstract

Recent reasoning language models, particularly those that employ long latent chains of thought, achieve strong performance on complex agentic tasks. However, as these models operate over increasingly long time horizons, their internal progress becomes opaque to users, making expectation management and real-time oversight difficult. In this work, we investigate whether real-time progress prediction is feasible for such models. We first test whether hidden states encode progress information by discretizing reasoning trajectories and training a linear probe to classify reasoning states. We then fine-tune models to generate progress estimates from 0--100\% during chain-of-thought reasoning. Our strongest progress-reporting checkpoint reaches 0.161 MAE on mathematical reasoning traces and outperforms position baselines in this setting. Finally, we quantify the intrinsic ambiguity of progress labels by measuring how much the implied progress value varies from the same partial rollout. This ambiguity is lowest for Qwen3-4B, whose continuations produce the smallest rollout dispersion, suggesting that larger models can make progress labels more stable by reducing variation in remaining solution length.
BibTeX
@misc{raaschoujensen2025predicting,
  author = {Hans Peter Lynsgøe Raaschou-jensen and Constanza Fierro and Anders Søgaard},
  title = {{Predicting thinking time in Reasoning models}},
  year = {2025},
  eprint = {2506.23274},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.23274},
}
Figure