Agent Memory as Rate-Limited Filtering: Explicit State Outlasts Long Context Beyond a Reader-Dependent Horizon

Tianhao Guo, Chongyao GuoarXiv 2026 · arXiv

TL;DR

Even the most advanced LLMs struggle to track "state" (who has access to what) as conversation or log histories grow, even when the information remains within their context window. This paper demonstrates that maintaining an explicit state table—updated via a "buildable filter"—is far more robust and efficient than relying on long-context reasoning or standard RAG for agentic tasks.

What Problem Does This Paper Address?

As AI agents are deployed to handle long-running tasks (like managing databases or monitoring system logs), they need a way to remember the current "truth" of the world. Traditionally, researchers have relied on two methods:

  1. Long Context: Feeding the entire history into the model.
  2. Retrieval (RAG): Pulling "relevant" snippets from the past.

The authors argue that both fail at state tracking. In many real-world scenarios—like access control—the answer depends on the order of events (e.g., a "Grant" followed by a "Withdrawal"). Standard retrieval is often "order-oblivious," and long contexts suffer from "dilution," where the model loses the decisive signal amidst hundreds of irrelevant log lines. The paper seeks to define the "operational horizon" where an LLM's ability to reason through a raw log collapses.

Methods

The researchers treat agent memory as a recursive filtering problem. Instead of viewing memory as a static pile of text, they view it as a state ($S_t$) that is updated every time a new observation ($O_t$) arrives.

The "Buildable Filter"

The authors propose a hybrid architecture:

  • The Parser: A small LLM reads a single log line and converts it into a structured JSON update (e.g., {"action": "withdraw", "user": "alice"}).
  • The Rule: A deterministic piece of code (the "transition rule") applies that update to a state table.
  • The Reader: When a question is asked, the LLM only reads the current state table, not the whole history.

Paired Counterfactuals

To test this, they created a benchmark using paired histories. Two logs are identical in wording and length, but the order of two key events is swapped (e.g., Log A ends in a grant; Log B ends in a withdrawal). If a model is truly tracking state, it should give opposite answers; if it's just "vibing" based on keywords, it will fail.

Datasets

  • Access Control Benchmark: A newly introduced executable benchmark involving grants, withdrawals, expiries, and source priorities (e.g., a "Security Office" command overrides an "Admin" command).
  • Second Domain (Customs Clearance): An independently specified domain involving shipments, holds, and clearances to ensure the findings weren't specific to the first task.
  • Models Evaluated: A wide range of models including Qwen2.5 (7B to 72B), phi-4, and massive reasoning models like gpt-oss-120b and DeepSeek-R1-Distill.

Key Insights and Ideas

  • The Operational Horizon: Every model has a limit. For a 14B model, accuracy on raw logs drops to chance (50%) after just 256 events. Even a 120B reasoning model hits a "horizon" around 1,024 events where an explicit state table begins to outperform it.
  • Order-Oblivious Failure: The paper mathematically proves that any memory system that ignores time order (like standard vector-based RAG or BM25) is doomed to a minimum error rate of ~50% on state-tracking tasks.
  • LLMs are Parsers, not Bookkeepers: Models are surprisingly good at parsing a single line correctly (99.9%+ accuracy). However, they are poor at maintaining a ledger over time when asked to rewrite it recursively; they tend to "erase" or drop inactive components that might become relevant later.
  • Prohibitions are "Sticky": Qualitatively, the authors found that models are more likely to remember a "Deny" or "Withdrawal" command than a "Reinstatement," leading to a "drift toward refusal" in long contexts.

Key Results

  • Performance Collapse: On the access log task, Qwen2.5-14B fell from 83% accuracy at 8 events to 48% (worse than chance) at 256 events.
  • State Restoration: When the same model was given an explicit state table instead of the log, its accuracy jumped back to 95-99%, with no decline as the history grew.
  • Filter Superiority: The "buildable filter" (LLM parser + code rule) achieved 80-89% accuracy at 256 events, significantly beating the Mem0 library, StateMem, and even long-context reasoning traces.
  • Efficiency: Reading a state table is roughly 6x cheaper in terms of tokens than reading a long log, and the cost doesn't grow with time.

Limitations and Future Work

  • Synthetic Constraints: The benchmarks use templates and a fixed update schema. Real-world logs may have more linguistic variety or cross-line references (e.g., "Do what he said").
  • Schema Dependency: The filter requires a predefined schema for what "state" looks like. If an agent encounters a completely new type of information, the filter might ignore it.
  • Future Direction: The authors suggest exploring "re-parsing" strategies—where a model only goes back to the raw log to re-verify specific keys when a query depends on them—as a way to correct persistent state errors.

Summary updated

Abstract

Can a language model that reads the full history of an access log say whether a service may act now? On an executable benchmark of grants, withdrawals, reinstatements, expiries, groups and change windows under a known policy, scored on pairs of histories that differ only in the order or one field of the deciding events, open models of 7 to 32 billion parameters fall from 72 to 86 percent after eight or sixteen events to 48 to 55 percent after 256, while reasoning-trained 32B models reading the current state reach 96 and 98 percent. A reasoning model of 120 billion parameters trails the state by four and seven points at 256 and 512 events and by about ten at 1,024, so the length beyond which explicit state outlasts long context grows with the reader. We treat agent memory as recursive filtering. The decision depends on the history only through a state whose size is set by the number of tracked keys. A store holding fewer component records than the state forgets on a time scale set by its capacity, and a reader model predicts its accuracy within 0.02 on average in a registered test. Retaining log lines without regard to time order errs with probability approaching one half, a bound that is attained. A filter in which a model parses each line and an explicit rule applies the updates answers 80 to 89 percent at 256 events and beats relevance retrieval, model-kept ledgers, a reimplementation of StateMem and the Mem0 library. A retriever that selects at query time the lines the queried state depends on matches it at 256 events on the first domain, so what decides is selection by the state's dependencies, which the filter performs when writing, without the query. Applying parsed updates themselves, the models fall to 58 and 59 percent. A frozen replication on a second domain with fresh wording reproduces the horizon.
BibTeX
@misc{guo2026agent,
  author = {Tianhao Guo and Chongyao Guo},
  title = {{Agent Memory as Rate-Limited Filtering: Explicit State Outlasts Long Context Beyond a Reader-Dependent Horizon}},
  year = {2026},
  eprint = {2611.02361},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.02361},
}
Figure