Measure Before You Manage: Evaluating Agent Working Memory in Coding Agents

Le Chen, Zishen Wan, Baixi Sun, Xiaolong Ma, Chih-Hsuan Yang, Feng Yan, Sheng Di, Franck Cappello, Rajeev ThakurarXiv 2026 · arXiv

TL;DR

LLM-based coding agents rely on a complex mix of memory objects—like source code, tool outputs, and instructions—that evolve differently over time. This paper demonstrates that evaluating memory management based purely on nominal token budgets is deeply misleading, and proposes a rigorous four-level framework to properly account for stored state, delivered context, management overhead, and task outcomes.

What Problem Does This Paper Address?

When AI agents tackle long-horizon coding tasks, their working memory quickly fills up with system instructions, source code files (artifacts), terminal outputs, and model-generated thoughts. Standard memory management techniques typically treat this state uniformly, using simple rules like First-In-First-Out (FIFO) or Least Recently Used (LRU) eviction once a token budget is reached.

However, agent memory is highly heterogeneous. Treating all tokens the same ignores their semantic roles and lifecycles. Furthermore, the authors identify a major evaluation gap: researchers frequently use a single "nominal token budget" to compare memory policies, which hides critical differences in actual delivered context, hidden computational costs, and true task performance.

Figure 1: Pooled object accounting across 55 coding trajectories. Each series has its own denominator: summed content volume or summed retention-weighted cost. These are descriptive shares, not task- level confidence intervals. The same four object types contribute different shares when residency is included.

Methods

The authors analyze agent working memory through a series of empirical studies and propose a new evaluation paradigm:

  1. Typed-Object Accounting: They categorize memory into four types (Tool output, Artifact, Instruction, Agent state) and track both their raw size (content volume) and how long they persist in the context window (retention-weighted cost).
  2. Semantically Informed Policies: They evaluate two advanced memory management strategies:
  3. Object-Aware Compression (OA): A heuristic policy that scores memory objects based on type, age, and access frequency, systematically demoting low-score items from raw text to compressed forms, summaries, or pointers.
  4. Retrieval-Based Management (GA): An adaptation of the recency/relevance/importance framework from the famous "Generative Agents" paper.
  5. Four-Level Evaluation Framework: To replace simplistic token-budget tracking, they introduce a structured reporting framework divided into four distinct layers:
  6. Stored State: The profile of what is kept in memory.
  7. Delivered Context: What actually gets sent to the LLM at each step.
  8. Management Work: The computational overhead (auxiliary LLM calls, embeddings) required to run the memory policy.
  9. Task/Process Outcome: The final impact on agent behavior and success.

Datasets

The study utilizes archived trajectories from two primary sources:

  • SWE-bench Lite Subset: 55 full-context trajectories across eight repositories, specifically filtered for tasks solvable under a local evaluator.
  • Terminal-Bench Probe: 18 trajectories used as a descriptive boundary check to see how memory composition shifts in a pure terminal environment.
Table 1: Study populations and comparison units. These samples must not be added together as independent replications. The retrieval study reuses development tasks rather than providing another held-out test.

Key Insights and Ideas

  • Volume $\neq$ Residency: Tool outputs make up the majority of raw memory volume (55.5%). However, because source code artifacts are retained in the context window much longer, their retention-weighted cost (38.9%) nearly matches that of tool outputs (40.2%).
  • The "Nominal Budget" Illusion: Setting a nominal budget (e.g., "15% of context") rarely reflects the actual pressure an agent experiences due to implementation details like budget floors. Two different runs under the same nominal cap can deliver drastically different amounts of actual context to the model.
  • Memory Management Isn't Free: Advanced memory policies introduce massive hidden costs. Summarizing old context or calculating embedding relevance requires auxiliary LLM calls and CPU/GPU time that are completely unaccounted for in standard token budgets.

Key Results

The authors found that apparent performance gains during a "calibration" (development) phase often fail to generalize to held-out tasks. As shown in the task-paired contrasts, the Object-Aware (OA) policy significantly reduced repeated tool calls during calibration but showed no statistically significant benefit on held-out data after Holm correction.

Table 2: Task-paired repeated-call contrasts: OA–baseline for calibration and held-out; GA–baseline for retrieval. Negative ∆favors OA or GA respectively; n/m: tasks/nonzero differences. CIs are post-hoc task bootstraps; exact two-sided signed-rank tests omit zero differences; Holm correction is separate for each of the three comparison families. The rows are not a cross-study ranking.

Furthermore, the resource accounting highlights the hidden "management work" required by complex policies. For example, over eight complete blocks, the retrieval-based policy (GA) forced 285 extra "importance" LLM calls, and the object-aware policy (OA) added 169 summary calls, significantly increasing the agent's total wall-time execution.

Table 3: Measured resources in the eight complete retrieval-study blocks. Main calls count recorded agent LLM steps; input is summed main-agent API prompt usage, excluding auxiliary requests; wall time is median whole-run seconds; ceiling counts runs reaching 25 steps, not successful repairs.

Limitations and Future Work

The authors explicitly note several limitations to their study. The evidence base is small and repository-clustered, consisting of only eight complete six-arm blocks for the retrieval follow-up. Because the data relies on historical archives, the served model revisions were unpinned, making exact trajectory regeneration impossible.

Additionally, the host system's instrumentation suffered from defects—such as automatically updating the "access clock" whenever an item was included in a prompt—which corrupted the recency signals used by the LRU and OA policies. The authors emphasize that future work must validate whether semantically informed memory management actually improves final task repair success, rather than just optimizing process regularity metrics.

Summary updated

Abstract

Agent working memory is heterogeneous. Objects such as instructions, artifacts, tool outputs, and agent-generated state play different semantic roles and exhibit different size, retention, and representation profiles. Recent work has begun to explore memory-management mechanisms that account for such heterogeneity. This work focuses on semantic heterogeneity and studies how it should shape the management and evaluation of working memory in coding agents. Across 55 archived coding-agent trajectories, we find that semantically different working-memory objects exhibit distinct retention and compression behavior. This heterogeneity motivates semantically informed memory management. We study two semantically informed strategies: an object-aware compression policy and a retrieval-based policy. Their evaluation shows that calibration gains may not transfer to held-out tasks, and that equal token budgets do not imply equal delivered context or management cost. A real-system replay further exposes serving limits that nominal budgets alone do not capture. Together, these results show why semantic structure matters for agent working memory and why evaluating memory-management strategies requires more than a nominal token budget. We organize these lessons into four levels: stored state, delivered context, management work, and task or process outcome.
BibTeX
@misc{chen2026measure,
  author = {Le Chen and Zishen Wan and Baixi Sun and Xiaolong Ma and Chih-Hsuan Yang and Feng Yan and Sheng Di and Franck Cappello and Rajeev Thakur},
  title = {{Measure Before You Manage: Evaluating Agent Working Memory in Coding Agents}},
  year = {2026},
  eprint = {2608.31057},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2608.31057},
}
Figure