Sleep-time Compute: Beyond Inference Scaling at Test-time

Kevin Lin, Charlie Snell, Yu Wang, Charles Packer, Sarah Wooders, Ion Stoica, Joseph E. GonzalezarXiv 2025 · DOI · OpenReview

Introduction

Scaling compute at inference time has emerged as a powerful paradigm for enhancing the reasoning capabilities of Large Language Models (LLMs). Models such as OpenAI’s o1 and DeepSeek-R1 demonstrate that allowing a model more "thinking time" during the generation process leads to significant performance gains on complex tasks. However, this approach—referred to as test-time compute—imposes a heavy burden on the end user in the form of high latency and increased operational costs. Because the reasoning happens entirely after a query is submitted, users often face wait times ranging from seconds to minutes.

This paper introduces sleep-time compute, a framework designed to alleviate these bottlenecks by shifting reasoning from the online interaction phase to an offline pre-computation phase. By leveraging the idle time before a user query arrives, models can pre-process known background context to generate enriched representations that streamline subsequent test-time reasoning.

The Sleep-time Compute Framework

The core innovation of this work is the bifurcation of the inference process into two distinct stages: Sleep-time and Test-time.

Comparison of standard test-time compute versus the proposed sleep-time compute framework, illustrating how pre-computed 'Learned Context' simplifies online answering.
  1. Sleep-time (Offline Phase): In many real-world applications, the background context is known before the user asks a specific question. Examples include a software repository, a long legal document, or a persistent chat history. During "sleep-time," the model is prompted to analyze this context $c$ without a specific query. It draws logical conclusions, identifies key relationships, and anticipates potential questions, resulting in a "Learned Context" $c'$.
  2. Test-time (Online Phase): When the user eventually submits a query $q$, the model is provided with the enriched context $c'$ instead of the raw context $c$. Because the heavy reasoning was performed during the sleep phase, the model can arrive at the correct answer using a significantly smaller test-time compute budget.

Experimental Methodology and Datasets

To evaluate the efficacy of sleep-time compute, the authors developed "stateful" versions of standard benchmarks. This involved decomposing problems into a static background context and a dynamic query.

Decomposition of a standard query into background context and a specific question for stateful evaluation.

The study utilized four primary datasets:

  • Stateful GSM-Symbolic & AIME: Math-heavy benchmarks where the problem setup (context) is separated from the final question (query).
  • Multi-Query GSM-Symbolic: A dataset designed to test cost amortization, where a single context is paired with multiple unique questions generated by o3-mini.
  • SWE-Features: A new agentic software engineering benchmark focusing on complex GitHub pull requests that span multiple files, requiring a deep understanding of repository-wide context.

Key Insights: Shifting the Pareto Frontier

The research yields several critical insights into how compute should be allocated in stateful systems. Most notably, sleep-time compute does not merely move the model along the existing accuracy-latency trade-off curve; it shifts the entire Pareto frontier outward. This means that for any given test-time latency target, sleep-time compute provides higher accuracy than standard test-time scaling.

Furthermore, the effectiveness of this approach is closely tied to query predictability. When the eventual user question can be reasonably inferred from the context, the gap between sleep-time and test-time performance widens. The authors also frame sleep-time compute as a form of natural language representation learning. Instead of optimizing vectors in a hidden layer, the model optimizes information within the space of natural language tokens, essentially "compressing" the reasoning required for future tasks.

Results and Performance Gains

The empirical results demonstrate substantial improvements in both efficiency and raw performance:

  • Compute Efficiency: Sleep-time compute achieved a ~5x reduction in the test-time compute required to reach a specific accuracy level on math benchmarks.
  • Accuracy Improvements: By scaling the reasoning effort during the sleep phase (e.g., using parallel agents), the authors observed accuracy boosts of up to 13% on GSM-Symbolic and 18% on AIME.
  • Cost Amortization: In scenarios where multiple queries are asked about the same context, the average cost per query was reduced by 2.5x, as the expensive sleep-time reasoning is only performed once.
  • Software Engineering: On the SWE-Features benchmark, the framework reduced test-time token usage by 1.5x while maintaining higher F1 scores compared to standard scaling methods.

Conclusion and Future Directions

Sleep-time compute represents a significant step toward making high-reasoning LLMs viable for low-latency applications. While the method's success depends on the predictability of user queries, it offers a clear path for optimizing stateful AI systems like coding assistants and document analysis tools.

Future research will focus on developing dynamic allocation strategies—systems that can automatically decide how much compute to spend during sleep-time versus test-time based on the complexity and predictability of the context. Additionally, the outputs of sleep-time reasoning could serve as high-quality synthetic data for pretraining the next generation of models, potentially embedding these reasoning shortcuts directly into the model weights.

Summary updated

Abstract

Scaling test-time compute has emerged as a key ingredient for enabling large language models (LLMs) to solve difficult problems, but comes with high latency and inference cost. We introduce sleep-time compute, which allows models to "think" offline about contexts before queries are presented: by anticipating what queries users might ask and pre-computing useful quantities, we can significantly reduce the compute requirements at test-time. To demonstrate the efficacy of our method, we create modified versions of two reasoning tasks - Stateful GSM-Symbolic and Stateful AIME. We find that sleep-time compute can reduce the amount of test-time compute needed to achieve the same accuracy by ~ 5x on Stateful GSM-Symbolic and Stateful AIME and that by scaling sleep-time compute we can further increase accuracy by up to 13% on Stateful GSM-Symbolic and 18% on Stateful AIME. Furthermore, we introduce Multi-Query GSM-Symbolic, which extends GSM-Symbolic by including multiple related queries per context. By amortizing sleep-time compute across related queries about the same context using Multi-Query GSM-Symbolic, we can decrease the average cost per query by 2.5x. We then conduct additional analysis to understand when sleep-time compute is most effective, finding the predictability of the user query to be well correlated with the efficacy of sleep-time compute. Finally, we conduct a case-study of applying sleep-time compute to a realistic agentic SWE task.
BibTeX
@misc{lin2025sleep,
  author = {Kevin Lin and Charlie Snell and Yu Wang and Charles Packer and Sarah Wooders and Ion Stoica and Joseph E. Gonzalez},
  title = {{Sleep-time Compute: Beyond Inference Scaling at Test-time}},
  year = {2025},
  eprint = {2504.13171},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2504.13171},
}
Figure