Can Computation from Earlier Problems Help LLMs Solve New Ones?

Jipei He, Wenhui Tan, Xiaoyi Yu, Enver Sangineto, Fiorenzo Parascandolo, Rita Cucchiara, Ruihua SongarXiv 2026 · arXiv

TL;DR

When Large Language Models (LLMs) solve a sequence of independent problems in one session, the "historical" computation from earlier turns can unexpectedly help or hinder their current performance. This paper introduces STAIR (Stale-Token Attention for Inter-query Reuse), a lightweight method that captures internal reasoning states from previous turns and uses a tiny, learned controller (just 12k parameters) to selectively reuse that computation, boosting reasoning accuracy by up to 11.67 percentage points.

What Problem Does This Paper Address?

In a typical multi-turn conversation, we often ask an LLM to solve several independent tasks in a row. While these tasks are logically separate, the LLM’s context window retains the entire history. The authors ask a fundamental question: Does the internal computational work the model did to solve Problem A affect its ability to solve Problem B?

Through preliminary experiments, they found that simply retaining the text of previous turns (the "Native" approach) is a double-edged sword. For some tasks, like graduate-level science (GPQA), history actually helps. For others, like math (MATH-500), it can degrade performance. The challenge is that the model's internal representations shift in response to history in complex ways. The authors sought a way to harness the useful parts of this "historical computation" while filtering out the noise, all without the massive cost of full-model fine-tuning.

Methods

The authors first performed a "crossed-history replay" to analyze how different conversation histories affect the representation of the same problem. They discovered that while history shifts a model's internal states, these shifts preserve a recurring, problem-dependent structure.

To exploit this, they developed STAIR (Figure 1).

STAIR overview in a continuous reasoning session

Key components include:

  1. Historical K/V Bank: As the model generates reasoning for earlier problems, STAIR captures the internal Key and Value (K/V) states and stores them in a read-only bank.
  2. Query-Side Re-addressing: When the model processes a new problem, a tiny controller learns to "reflect" the current queries. Using Householder reflections, the controller reverses a specific component of the query vector, changing its direction (and thus what it attends to in the bank) without changing its length.
  3. Differential Readout: STAIR calculates the difference between a "reference" read of the bank and the "reflected" read. This differential is then added back to the model's standard attention output.
  4. Frozen Backbone: Crucially, the LLM itself remains frozen. Only the reflection directions (12,288 parameters total) are trained, making it extremely parameter-efficient.
The STAIR architecture within a transformer block

Datasets

  • Training: The STAIR controllers were trained on DAPO-Math-17k, a dataset of mathematical reasoning problems.
  • Evaluation: The authors tested the method across four major benchmarks:
  • MATH-500: Standard high-school math competition problems.
  • AIME 2025: Advanced American Invitational Mathematics Examination problems.
  • AMC23: American Mathematics Competitions problems.
  • GPQA-Diamond: Graduate-level, "Google-proof" science questions, used to test out-of-domain transfer.

Key Insights and Ideas

  • Computation is a Resource: The paper argues that we should treat the internal states generated during reasoning not just as transient data, but as a reusable resource for subsequent tasks.
  • Attention Mass Persistence: The authors found that even in middle and later layers of the transformer, the model continues to allocate significant "attention mass" to earlier assistant responses (Figure 2B).
  • Structural Consistency: Even though different histories shift a problem's representation, the "neighborhood" of similar problems in the latent space remains remarkably stable (Figure 2C). This suggests that the model's response to history is organized, not chaotic.
Analysis of attention mass and state displacement across layers

Key Results

STAIR showed significant improvements over unmodified models (Native) across multiple Qwen model sizes (4B and 9B):

  • AIME 2025: The most dramatic gain was seen here, where STAIR improved the Avg@4 accuracy of Qwen3.5-4B by 11.67 percentage points over the Native model.
  • MATH-500: STAIR allowed Qwen3.5 models to actually exceed their "Vanilla" (no history) performance, effectively turning the conversation history into a net positive.
  • Complementary to LoRA: When trained alongside LoRA (Low-Rank Adaptation), STAIR provided an additional boost, suggesting that learning to read history is a distinct and complementary skill to adapting model weights.

Limitations and Future Work

  • Domain Transfer: While STAIR worked well for math, the results on GPQA-Diamond were mixed for some models. This suggests that the "reading rules" learned for math might not perfectly translate to scientific reasoning.
  • Memory Overhead: Maintaining the K/V bank is not free. For long sessions, the bank can grow quite large (the authors noted a peak memory increase of ~6.6 GiB for a full bank), which might necessitate selective pruning or compression in the future.
  • Session Length: The current study focused on four-turn sessions. Exploring how these effects evolve over dozens of turns remains an open question.

Summary updated

Abstract

Large language models often solve independent problems in the same conversation. Can computation from earlier problems help them solve new ones? To answer this question, we first conduct preliminary experiments showing that retained history can raise or lower later-turn accuracy, even within the same domain. To understand these effects, we use controlled replay to isolate internal state changes specific to each problem-history pairing. Across different histories, these changes preserve similar relationships among current problems. To improve reasoning under retained history, we introduce STAIR (Stale-Token Attention for Inter-query Reuse). STAIR captures keys and values from earlier response generation in a fixed bank. It learns to redirect current queries when they read this bank during prompt processing. The base model remains frozen; only 12,288 parameters are trained. Across three Qwen models and four benchmarks, STAIR improves average later-turn accuracy by up to 11.67 percentage points over the unmodified model with history.
BibTeX
@misc{he2026can,
  author = {Jipei He and Wenhui Tan and Xiaoyi Yu and Enver Sangineto and Fiorenzo Parascandolo and Rita Cucchiara and Ruihua Song},
  title = {{Can Computation from Earlier Problems Help LLMs Solve New Ones?}},
  year = {2026},
  eprint = {2609.39394},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2609.39394},
}
Figure