TL;DR
Even the most advanced LLMs struggle to track "state" (who has access to what) as conversation or log histories grow, even when the information remains within their context window. This paper demonstrates that maintaining an explicit state table—updated via a "buildable filter"—is far more robust and efficient than relying on long-context reasoning or standard RAG for agentic tasks.
What Problem Does This Paper Address?
As AI agents are deployed to handle long-running tasks (like managing databases or monitoring system logs), they need a way to remember the current "truth" of the world. Traditionally, researchers have relied on two methods:
- Long Context: Feeding the entire history into the model.
- Retrieval (RAG): Pulling "relevant" snippets from the past.
The authors argue that both fail at state tracking. In many real-world scenarios—like access control—the answer depends on the order of events (e.g., a "Grant" followed by a "Withdrawal"). Standard retrieval is often "order-oblivious," and long contexts suffer from "dilution," where the model loses the decisive signal amidst hundreds of irrelevant log lines. The paper seeks to define the "operational horizon" where an LLM's ability to reason through a raw log collapses.
Methods
The researchers treat agent memory as a recursive filtering problem. Instead of viewing memory as a static pile of text, they view it as a state ($S_t$) that is updated every time a new observation ($O_t$) arrives.
The "Buildable Filter"
The authors propose a hybrid architecture:
- The Parser: A small LLM reads a single log line and converts it into a structured JSON update (e.g.,
{"action": "withdraw", "user": "alice"}). - The Rule: A deterministic piece of code (the "transition rule") applies that update to a state table.
- The Reader: When a question is asked, the LLM only reads the current state table, not the whole history.
Paired Counterfactuals
To test this, they created a benchmark using paired histories. Two logs are identical in wording and length, but the order of two key events is swapped (e.g., Log A ends in a grant; Log B ends in a withdrawal). If a model is truly tracking state, it should give opposite answers; if it's just "vibing" based on keywords, it will fail.
Datasets
- Access Control Benchmark: A newly introduced executable benchmark involving grants, withdrawals, expiries, and source priorities (e.g., a "Security Office" command overrides an "Admin" command).
- Second Domain (Customs Clearance): An independently specified domain involving shipments, holds, and clearances to ensure the findings weren't specific to the first task.
- Models Evaluated: A wide range of models including Qwen2.5 (7B to 72B), phi-4, and massive reasoning models like gpt-oss-120b and DeepSeek-R1-Distill.
Key Insights and Ideas
- The Operational Horizon: Every model has a limit. For a 14B model, accuracy on raw logs drops to chance (50%) after just 256 events. Even a 120B reasoning model hits a "horizon" around 1,024 events where an explicit state table begins to outperform it.
- Order-Oblivious Failure: The paper mathematically proves that any memory system that ignores time order (like standard vector-based RAG or BM25) is doomed to a minimum error rate of ~50% on state-tracking tasks.
- LLMs are Parsers, not Bookkeepers: Models are surprisingly good at parsing a single line correctly (99.9%+ accuracy). However, they are poor at maintaining a ledger over time when asked to rewrite it recursively; they tend to "erase" or drop inactive components that might become relevant later.
- Prohibitions are "Sticky": Qualitatively, the authors found that models are more likely to remember a "Deny" or "Withdrawal" command than a "Reinstatement," leading to a "drift toward refusal" in long contexts.
Key Results
- Performance Collapse: On the access log task, Qwen2.5-14B fell from 83% accuracy at 8 events to 48% (worse than chance) at 256 events.
- State Restoration: When the same model was given an explicit state table instead of the log, its accuracy jumped back to 95-99%, with no decline as the history grew.
- Filter Superiority: The "buildable filter" (LLM parser + code rule) achieved 80-89% accuracy at 256 events, significantly beating the Mem0 library, StateMem, and even long-context reasoning traces.
- Efficiency: Reading a state table is roughly 6x cheaper in terms of tokens than reading a long log, and the cost doesn't grow with time.
Limitations and Future Work
- Synthetic Constraints: The benchmarks use templates and a fixed update schema. Real-world logs may have more linguistic variety or cross-line references (e.g., "Do what he said").
- Schema Dependency: The filter requires a predefined schema for what "state" looks like. If an agent encounters a completely new type of information, the filter might ignore it.
- Future Direction: The authors suggest exploring "re-parsing" strategies—where a model only goes back to the raw log to re-verify specific keys when a query depends on them—as a way to correct persistent state errors.
Summary updated