TL;DR
LLM-based coding agents rely on a complex mix of memory objects—like source code, tool outputs, and instructions—that evolve differently over time. This paper demonstrates that evaluating memory management based purely on nominal token budgets is deeply misleading, and proposes a rigorous four-level framework to properly account for stored state, delivered context, management overhead, and task outcomes.
What Problem Does This Paper Address?
When AI agents tackle long-horizon coding tasks, their working memory quickly fills up with system instructions, source code files (artifacts), terminal outputs, and model-generated thoughts. Standard memory management techniques typically treat this state uniformly, using simple rules like First-In-First-Out (FIFO) or Least Recently Used (LRU) eviction once a token budget is reached.
However, agent memory is highly heterogeneous. Treating all tokens the same ignores their semantic roles and lifecycles. Furthermore, the authors identify a major evaluation gap: researchers frequently use a single "nominal token budget" to compare memory policies, which hides critical differences in actual delivered context, hidden computational costs, and true task performance.
Methods
The authors analyze agent working memory through a series of empirical studies and propose a new evaluation paradigm:
- Typed-Object Accounting: They categorize memory into four types (Tool output, Artifact, Instruction, Agent state) and track both their raw size (content volume) and how long they persist in the context window (retention-weighted cost).
- Semantically Informed Policies: They evaluate two advanced memory management strategies:
- Object-Aware Compression (OA): A heuristic policy that scores memory objects based on type, age, and access frequency, systematically demoting low-score items from raw text to compressed forms, summaries, or pointers.
- Retrieval-Based Management (GA): An adaptation of the recency/relevance/importance framework from the famous "Generative Agents" paper.
- Four-Level Evaluation Framework: To replace simplistic token-budget tracking, they introduce a structured reporting framework divided into four distinct layers:
- Stored State: The profile of what is kept in memory.
- Delivered Context: What actually gets sent to the LLM at each step.
- Management Work: The computational overhead (auxiliary LLM calls, embeddings) required to run the memory policy.
- Task/Process Outcome: The final impact on agent behavior and success.
Datasets
The study utilizes archived trajectories from two primary sources:
- SWE-bench Lite Subset: 55 full-context trajectories across eight repositories, specifically filtered for tasks solvable under a local evaluator.
- Terminal-Bench Probe: 18 trajectories used as a descriptive boundary check to see how memory composition shifts in a pure terminal environment.
Key Insights and Ideas
- Volume $\neq$ Residency: Tool outputs make up the majority of raw memory volume (55.5%). However, because source code artifacts are retained in the context window much longer, their retention-weighted cost (38.9%) nearly matches that of tool outputs (40.2%).
- The "Nominal Budget" Illusion: Setting a nominal budget (e.g., "15% of context") rarely reflects the actual pressure an agent experiences due to implementation details like budget floors. Two different runs under the same nominal cap can deliver drastically different amounts of actual context to the model.
- Memory Management Isn't Free: Advanced memory policies introduce massive hidden costs. Summarizing old context or calculating embedding relevance requires auxiliary LLM calls and CPU/GPU time that are completely unaccounted for in standard token budgets.
Key Results
The authors found that apparent performance gains during a "calibration" (development) phase often fail to generalize to held-out tasks. As shown in the task-paired contrasts, the Object-Aware (OA) policy significantly reduced repeated tool calls during calibration but showed no statistically significant benefit on held-out data after Holm correction.
Furthermore, the resource accounting highlights the hidden "management work" required by complex policies. For example, over eight complete blocks, the retrieval-based policy (GA) forced 285 extra "importance" LLM calls, and the object-aware policy (OA) added 169 summary calls, significantly increasing the agent's total wall-time execution.
Limitations and Future Work
The authors explicitly note several limitations to their study. The evidence base is small and repository-clustered, consisting of only eight complete six-arm blocks for the retrieval follow-up. Because the data relies on historical archives, the served model revisions were unpinned, making exact trajectory regeneration impossible.
Additionally, the host system's instrumentation suffered from defects—such as automatically updating the "access clock" whenever an item was included in a prompt—which corrupted the recency signals used by the LRU and OA policies. The authors emphasize that future work must validate whether semantically informed memory management actually improves final task repair success, rather than just optimizing process regularity metrics.
Summary updated