TL;DR
When Large Language Models (LLMs) solve a sequence of independent problems in one session, the "historical" computation from earlier turns can unexpectedly help or hinder their current performance. This paper introduces STAIR (Stale-Token Attention for Inter-query Reuse), a lightweight method that captures internal reasoning states from previous turns and uses a tiny, learned controller (just 12k parameters) to selectively reuse that computation, boosting reasoning accuracy by up to 11.67 percentage points.
What Problem Does This Paper Address?
In a typical multi-turn conversation, we often ask an LLM to solve several independent tasks in a row. While these tasks are logically separate, the LLM’s context window retains the entire history. The authors ask a fundamental question: Does the internal computational work the model did to solve Problem A affect its ability to solve Problem B?
Through preliminary experiments, they found that simply retaining the text of previous turns (the "Native" approach) is a double-edged sword. For some tasks, like graduate-level science (GPQA), history actually helps. For others, like math (MATH-500), it can degrade performance. The challenge is that the model's internal representations shift in response to history in complex ways. The authors sought a way to harness the useful parts of this "historical computation" while filtering out the noise, all without the massive cost of full-model fine-tuning.
Methods
The authors first performed a "crossed-history replay" to analyze how different conversation histories affect the representation of the same problem. They discovered that while history shifts a model's internal states, these shifts preserve a recurring, problem-dependent structure.
To exploit this, they developed STAIR (Figure 1).
Key components include:
- Historical K/V Bank: As the model generates reasoning for earlier problems, STAIR captures the internal Key and Value (K/V) states and stores them in a read-only bank.
- Query-Side Re-addressing: When the model processes a new problem, a tiny controller learns to "reflect" the current queries. Using Householder reflections, the controller reverses a specific component of the query vector, changing its direction (and thus what it attends to in the bank) without changing its length.
- Differential Readout: STAIR calculates the difference between a "reference" read of the bank and the "reflected" read. This differential is then added back to the model's standard attention output.
- Frozen Backbone: Crucially, the LLM itself remains frozen. Only the reflection directions (12,288 parameters total) are trained, making it extremely parameter-efficient.
Datasets
- Training: The STAIR controllers were trained on DAPO-Math-17k, a dataset of mathematical reasoning problems.
- Evaluation: The authors tested the method across four major benchmarks:
- MATH-500: Standard high-school math competition problems.
- AIME 2025: Advanced American Invitational Mathematics Examination problems.
- AMC23: American Mathematics Competitions problems.
- GPQA-Diamond: Graduate-level, "Google-proof" science questions, used to test out-of-domain transfer.
Key Insights and Ideas
- Computation is a Resource: The paper argues that we should treat the internal states generated during reasoning not just as transient data, but as a reusable resource for subsequent tasks.
- Attention Mass Persistence: The authors found that even in middle and later layers of the transformer, the model continues to allocate significant "attention mass" to earlier assistant responses (Figure 2B).
- Structural Consistency: Even though different histories shift a problem's representation, the "neighborhood" of similar problems in the latent space remains remarkably stable (Figure 2C). This suggests that the model's response to history is organized, not chaotic.
Key Results
STAIR showed significant improvements over unmodified models (Native) across multiple Qwen model sizes (4B and 9B):
- AIME 2025: The most dramatic gain was seen here, where STAIR improved the
Avg@4accuracy of Qwen3.5-4B by 11.67 percentage points over the Native model. - MATH-500: STAIR allowed Qwen3.5 models to actually exceed their "Vanilla" (no history) performance, effectively turning the conversation history into a net positive.
- Complementary to LoRA: When trained alongside LoRA (Low-Rank Adaptation), STAIR provided an additional boost, suggesting that learning to read history is a distinct and complementary skill to adapting model weights.
Limitations and Future Work
- Domain Transfer: While STAIR worked well for math, the results on GPQA-Diamond were mixed for some models. This suggests that the "reading rules" learned for math might not perfectly translate to scientific reasoning.
- Memory Overhead: Maintaining the K/V bank is not free. For long sessions, the bank can grow quite large (the authors noted a peak memory increase of ~6.6 GiB for a full bank), which might necessitate selective pruning or compression in the future.
- Session Length: The current study focused on four-turn sessions. Exploring how these effects evolve over dozens of turns remains an open question.
Summary updated