TL;DR
Retaining full conversation history in multi-turn LLM interactions causes a massive collapse in reasoning accuracy—dropping from 60% to 10% within just twenty turns—because models "anchor" on their own earlier mistakes. Crucially, this failure is "silent": standard quality metrics like logical coherence and consistency remain high or even improve as the model becomes confidently wrong, making the degradation nearly impossible to detect in real-time without ground-truth labels.
What Problem Does This Paper Address?
As LLMs are integrated into complex decision-support systems (like logistics assistants or procurement trackers), they must maintain state over many turns. Traditionally, developers simply feed the entire transcript back into the model's context.
The author identifies two major gaps in our understanding of this practice:
- The Measurement Gap: Most benchmarks only test a few turns. We don't know how reasoning holds up over long horizons (100+ turns) when the model is forced to build upon its own previous reasoning.
- The Monitoring Gap: In a deployed system, you don't have the "gold" (correct) answer. Can we use label-free metrics (like how consistent the model is with itself) to detect that a model is starting to "drift" or fail before the final answer is wrong?
The core concern is a reliability trap: if a model makes an error at turn 5, it might generate turn 6 perfectly consistently with that error. To an observer, the model looks fluent and logical, but it is actually hallucinating a false task state.
Methods
The paper employs two distinct studies to isolate the "cost" of history:
- Study 1 (Short Horizon): Uses "sharded" math problems where information is revealed turn-by-turn. It compares a Dialogue History condition (full transcript) against a Fresh Prefix condition (only the user's information shards, no previous model replies). This isolates the effect of the model seeing its own previous words.
- Study 2 (Long Horizon): Introduces a 100-turn state-tracking challenge. It compares Dialogue History against a Verified State Summary (where a symbolic solver provides a ground-truth "ledger" of the current state, replacing the messy transcript).
- Monitoring Framework: The author tested six "RQEval" dimensions (Accuracy, Consistency, Robustness, Logical Coherence, Efficiency, and Stability) alongside five standard statistical drift detectors (like CUSUM and ADWIN) to see if any could serve as an "early warning" system.
Datasets
- Lost in Conversation (Math subset): Derived from GSM8K, these are arithmetic word problems split into shards.
- CDR-Stream (New Benchmark): A purpose-built benchmark of 100-turn dialogues across four domains: budget ledgers, warehouse inventory, lab protocols, and project milestones. It features an exact "gold" answer for every turn and a 12-level difficulty ladder that is calibrated specifically for each model tested (GPT-4o-mini, Llama-3 70B, and Gemma 3 12B).
Key Insights and Ideas
- The Anchoring Effect: Models are highly susceptible to "self-conditioning." Once an LLM commits to a wrong state in the transcript, it treats that error as a factual anchor for all future reasoning.
- The Stability Paradox: Perhaps the most surprising finding is that as accuracy collapses, stability actually rises. Because the model is anchoring on its own previous context, its repeated samples become more similar to each other. A "stable" model is not necessarily a correct one; it might just be consistently wrong.
- Coherence is Not Correctness: Logical coherence (the absence of contradictions between steps) fell far less than accuracy. A model can be perfectly "logical" while operating on a completely false premise inherited from turn 10.
- The Power Problem: Detecting drift in a single 100-turn conversation is statistically difficult. The author shows that a single stream only has enough "power" to detect massive accuracy drops; smaller, insidious degradations require pooling data across many sessions.
Key Results
- Accuracy Collapse: In Study 2, accuracy under full dialogue history plummeted from ~0.6 to ~0.1 within the first twenty turns.
- The Mitigation Works: Replacing the transcript with a verified state summary kept accuracy 0.52–0.59 higher and allowed models to recover from difficult turns—a feat the "history" models never achieved.
- Study 1 Impact: Even in short math tasks (median 6 turns), simply seeing its own previous replies lowered success by 0.21 for GPT-4o-mini and 0.29 for Mistral-7B.
- Silent Failure: None of the label-free metrics (Consistency, Coherence, Stability) provided a robust early warning. They often moved in the "wrong" direction or lagged behind the actual collapse in accuracy.
Limitations and Future Work
- Task Narrowness: The study focused on arithmetic and state-tracking with exact answers. It remains to be seen if these "silent costs" apply similarly to creative or open-ended prose generation.
- Model Selection: The models tested were capable but not the absolute "frontier" (e.g., GPT-4o-mini vs. GPT-4o). Stronger models might have a longer "grace period" before the collapse.
- Future Directions: The author suggests moving away from raw transcripts as memory. Instead, systems should maintain explicit "state files" (like a JSON ledger) that are updated and verified outside the LLM's primary reasoning loop. They also suggest exploring "counterfactual re-queries"—asking the model the same question without the history to see if the answer changes.
Summary updated