How Much Should an Agent Remember? Deployment Lessons from Self-Distilled Memory for LLM Agents

K. R. Jayaram, Vatche Isahagian, Aswath Vaithinathan Aravindan, Vinod Muthusamy, Punleuk Oum, Gaodan Fang, Michal Ulewicz, Gegi Thomas, Evelyn DuesterwaldarXiv 2026 · arXiv

TL;DR

LLM agents often fail not because they lack knowledge, but because they make recurring procedural mistakes. This paper introduces a self-distilled memory pipeline that allows agents to learn from their own history, revealing a crucial deployment lesson: the optimal "dose" of memory is a property of the model's capability—strong models thrive on more context, while weaker models require surgical selection to avoid being overwhelmed.

What Problem Does This Paper Address?

Even the most capable LLM agents struggle with "long-horizon" tasks—complex workflows requiring multiple steps. Interestingly, their failures are rarely due to a lack of information. An agent might know an API exists but fail to paginate it correctly, or it might resolve the wrong contact despite having access to the right one. These are procedural lapses, and because they recur, they are learnable.

While the idea of "agentic memory" isn't new, the authors argue that we lack a rigorous account of what happens when these systems are actually deployed and measured across different model tiers. They aim to move beyond "it works" to "here is how it fails and how to fix it."

Methods

The authors developed an open-source library called altk-evolve that implements a three-stage memory pipeline:

  1. Extraction: After an agent completes a task (success or failure), an LLM mines one to three short, imperative "lessons" from the trajectory (e.g., "Always check if the API returned a 'next_page' token").
  2. Consolidation: To prevent the memory from becoming a bloated list of near-duplicates, the system clusters related lessons into "guidelines." Crucially, this process is support-conserving: each guideline tracks how many raw lessons were merged into it.
  3. Delivery: At inference time, guidelines are either injected statically into the prompt or retrieved dynamically based on the current task's similarity to past experiences.

Datasets

The primary evaluation used AppWorld, a robust benchmark simulating nine applications (email, payments, etc.) via 457 APIs. It provides authoritative pass/fail verdicts and graded difficulty. The authors also validated their findings on $\tau$-bench, a customer-service benchmark, and through real-world deployment in IBM’s CUGA agent and watsonx Orchestrate platform.

Key Insights and Ideas

The paper is structured around seven "Insights" gained during deployment:

  • Memory "Dose" is Model-Dependent: This is the central finding. As shown in Figure 1, a strong model like DeepSeek-V3.2 continues to improve as you add more guidelines. However, a weaker model like gpt-oss-120b actually performs worse with a full memory set than with a small, carefully selected subset.
Figure 1: Dose–response on test_normal. Left: a strong model keeps improving as the injected guideline set grows from 0 to all 191 (“retr.” = 47 high-support guidelines plus 10 retrieved per task). Adjacent points differ by two to four tasks, within noise. The endpoints differ by 16. Right: a weak model gains most from about 29 selected guidelines (mean of two runs, bar shows both). All 199 are no better than a 24-guideline core.
  • The Value is in the Long Tail: Many systems try to compress memory into a tidy "top 10" list. The authors found this discards most of the value. The real performance gains come from "one-off" situational lessons that only appear once in the history but are vital when that specific situation recurs.
  • Recurrence $\neq$ Correctness: A "high-support" guideline (one derived from many episodes) can still be wrong. If an agent consistently misunderstands an API's default behavior, it will generate a "popular" but incorrect lesson. Memory systems need a signal for correctness or recency, not just frequency.
  • Silent Failures are the Real Enemy: In production, the biggest hurdles weren't modeling issues but "unglamorous" bugs: swallowed HTTP errors, idle network connections, or write-hooks that never fired. They recommend "failing loudly" on any per-task exception.

Key Results

  • DeepSeek-V3.2 (Strong): Gained +9.5 points in task completion (TGC) and +16.1 points in scenario completion (SGC) using its full 191-guideline memory set.
  • gpt-oss-120b (Weak): While the full set provided a boost, a selected subset of ~29 guidelines yielded the best result, outperforming the full set by 7 tasks.
  • Efficiency: For capable models, the "sweet spot" was often a subset of guidelines with at least two supporting lessons, providing significant gains with only a ~27% increase in token cost.
  • Qualitative Win: In a real-world expense-claim task, the memory system allowed an agent to "remember" to convert a receipt format before submitting, reducing a 13-step process to 10 steps and cutting latency by 39% (see Figure 3).
Figure 3: Cross-task transfer in the expense-claim workflow. Top: the source task invokes submit_expense_claim twice (13 steps, 18.53 s). Repeated calls can be dangerous when tools have expensive side effects, as in finance or insurance. Bottom: on a different task, a critical-memory node retrieves the learned procedure. The assistant converts the TIFF receipt before submitting once (10 steps, 11.35 s), an observed latency reduction of approximately 39%. This demonstrates transfer across related task instances. The timing difference is descriptive, not a controlled estimate.

Limitations and Future Work

The authors acknowledge that their current pipeline lacks a robust way to handle contradictory or stale memory. If an API changes, the agent might still be following "high-support" guidelines that are now obsolete. They suggest that future work should integrate "recency" or "correctness" signals—perhaps allowing the model to "down-vote" or delete harmful insights. Additionally, they note that run-to-run noise in LLM evaluations is high enough to fabricate "wins," necessitating repeated runs to establish a true noise floor.

Summary updated

Abstract

LLM agents on long-horizon tasks fail in recurring, procedural ways. They mis-paginate an API, resolve the wrong entity, or answer when no answer was asked. We built a pipeline that lets an agent learn from its own history. It first mines short \emph{lessons} from the agent's own trajectories, one to three per episode. It then consolidates those lessons into a smaller set of \emph{guidelines}, where each guideline records how many lessons it merged. At inference the agent receives either all guidelines or a selected subset. The pipeline ships in the open-source altk-evolve library and is deployed in IBM's CUGA agent and watsonx Orchestrate platform. We report what we learned building, measuring, and productizing it. Our primary instrument is the AppWorld benchmark, across models spanning 40--92\% baseline task completion. The central finding is that the right amount of memory is a property of the model, not the memory. A strong model keeps improving as guidelines are added. DeepSeek-V3.2 gains +9.5 points of task completion and +16.1 points of scenario completion with its full 191-guideline set. A weak model gains less from the same full set than from a small selected subset. gpt-oss-120b starts at 39.9\% task completion. It reaches 51.2\% with all 199 of its guidelines but 55.4\% (mean of two runs) with a selected subset of about 29. Around that result we report what went wrong: a high-support guideline that backfired, provenance silently dropped during consolidation, a swallowed API error that failed every task while producing a plausible log, and a baseline confound that nearly produced a misleading comparison with a prior method. We distill these into recommendations for teams deploying agent memory.
BibTeX
@misc{jayaram2026how,
  author = {K. R. Jayaram and Vatche Isahagian and Aswath Vaithinathan Aravindan and Vinod Muthusamy and Punleuk Oum and Gaodan Fang and Michal Ulewicz and Gegi Thomas and Evelyn Duesterwald},
  title = {{How Much Should an Agent Remember? Deployment Lessons from Self-Distilled Memory for LLM Agents}},
  year = {2026},
  eprint = {2611.26361},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.26361},
}
Figure