TL;DR
When AI reasoning models lose their previous "thoughts" (reasoning traces) during multi-turn tasks—a common side effect of modern API routing—some models barely flinch while others collapse entirely. This paper introduces "deletion sensitivity" as a measurable model-level property, revealing that model size is a poor predictor of how well an AI handles a "stripped" history.
What Problem Does This Paper Address?
Modern reasoning models (like DeepSeek-R1 or OpenAI’s o1) don't just output answers; they generate internal "reasoning traces" to plan and execute complex, multi-turn tool-calling sessions. However, in the real-world LLM ecosystem, these traces are often fragile. When a request is routed across different service providers or aggregators, the "thought" history is frequently stripped for security or billing reasons, leaving the model with only the raw record of previous actions and results.
Until now, the actual performance cost of this "history stripping" was unknown. Does losing the "why" behind previous steps cause models to lose their way? The authors set out to quantify this cost, which they term deletion sensitivity.
Methods
The researchers developed a rigorous, blind, multi-seed framework to measure how models behave when their reasoning history is deleted. They tested seven open-weight models under three specific rendering conditions:
- C1 (Retention): The "gold standard" where the full reasoning history is kept.
- C2 (Cross-turn Deletion): All historical reasoning traces are stripped, leaving only actions and results.
- C3 (Mid-session Deletion): History is stripped up to a specific point (40% of the session) to simulate a mid-task routing switch.
The study used the official Berkeley Function Calling Leaderboard (BFCL) checker to grade outcomes blindly, ensuring that the evaluation was independent of the experimental conditions. They tracked two primary indicators: Success Loss ($\Delta_{del}(S)$) and Repetition Rise ($\Delta_{del}(R)$).
Datasets
The evaluation utilized the BFCL v4 multi-turn subset, consisting of 794 tasks ranging from 2 to 8 turns. These tasks involve real tool environments, long contexts, and scenarios with missing parameters or functions, providing a challenging "stress test" for multi-turn reasoning.
Key Insights and Ideas
- Sensitivity is a Model Property, Not a Size Property: Surprisingly, parameter count does not predict how a model handles deletion. In the 32B parameter class alone, the performance drop spanned a massive 51.5 points. Some models are inherently "deletion-tolerant," while others are "deletion-sensitive."
- Three Behavioral Signatures of Failure: The authors identified three distinct ways models react to losing their history:
- Compensating: These models (like the Qwen3 family) simply re-derive their lost thoughts. They might issue more tool calls to "catch up," but they ultimately succeed.
- Looping: Models get stuck in a "Groundhog Day" loop, re-issuing the same tool call because they've forgotten they already did it (e.g., Olmo-3-7B-Think).
- Silent Failure: The model fails to reach the goal but doesn't show obvious signs of distress or repetition in the logs (e.g., Baichuan-M2-32B).
- Dose Invariance: For tolerant models, the amount of history deleted (25% vs. 100%) barely matters. If they can compensate, they can do it regardless of how much was stripped.
Key Results
The impact of deletion is dramatic and highly variable across models:
- The Performance Gap: Success loss ranged from -4.4 points (where deletion actually improved the R1-Distill-14B model by removing "trace baggage") to a staggering +51.3 points for Baichuan-M2-32B, which effectively collapsed without its history.
- The Qwen3 "Anchor": The Qwen3 family (8B, 14B, and 32B) proved remarkably stable, with success rates moving by less than 3.1 points across all conditions.
- Repetition Explosion: For "looping" models, repetition rates skyrocketed. Olmo-3-7B-Think saw its repetition rate jump from 30.2% to 66.5% when history was stripped.
Limitations and Future Work
The authors acknowledge several areas for further exploration:
- Task Specificity: The current findings are established on function-calling tasks. Whether these sensitivities transfer to other domains (like creative writing or mathematical proof) remains an open question.
- Closed Models: Due to the need for "white-box" control over the message layer, the study focused on open-weight models. Testing closed models (like GPT-4o or Claude 3.5) would require deeper API-level cooperation.
- Routing Intelligence: The paper suggests that "deletion sensitivity" should be a standard metric in model cards, allowing routers to make informed decisions about when it is "safe" to strip history to save on context costs. Developing these automated routing policies is a logical next step.
Summary updated