What Is This Memory Useful For? Measuring Task-Conditioned Utility in Long-Term Agent Memory

Chih-Hsuan Yang, Anjir Ahmed Chowdhury, Feng Yan, Rajeev ThakurarXiv 2026 · arXiv

TL;DR

This paper argues that a memory's value isn't a single "importance score" but depends entirely on the specific future task it's asked to help with. By introducing a "task-conditioned" evaluation protocol, the authors demonstrate that common memory compression techniques (like summarization) often "summarize away" the very details needed for factual recall, making raw text surprisingly hard to beat.

What Problem Does This Paper Address?

As LLM agents tackle longer horizons, they face a "memory management" problem: out of thousands of past experiences, which should they keep (retention), which should they look for (retrieval), and which should they actually show the model (exposure)?

Current systems often assign a single importance score to a memory based on recency or a generic "saliency" check. However, a memory of a budget spreadsheet is incredibly useful for a question about finances but useless for a question about software architecture. The authors tackle the challenge of measuring this task-conditioned utility—the specific delta a memory provides for a specific task—while keeping the memory "writer" blind to what those future tasks might be.

Methods

The authors propose a Conditional Memory Utility Protocol designed to isolate the effect of a single memory unit.

  1. Task-Blind Writing: A "writer" model processes a past experience (e.g., a page visit) into a representation (Raw, Summary, or Structured) without knowing any future questions.
  2. Counterfactual Evaluation: A "frozen reader" model answers a task twice: once with the target memory and once without it. Crucially, both versions include the same "filler" context to ensure the model isn't just reacting to a longer prompt.
  3. The Metric ($\Delta U$): Utility is measured as the change in reward (accuracy) between the "with-memory" and "without-memory" arms.
  4. Answer-Bearing Screen: To avoid drowning in "noise" (since most memories are irrelevant to most tasks), they use a screening tool to identify "answer-bearing" pairs—cases where the memory actually contains the information needed for the gold answer.
The evaluation protocol: (a) task-blind writing, (b) counterfactual comparison, and (c) building a task-wise utility profile.

Datasets

  • LongMemEval-V2: The primary benchmark, consisting of web-agent and enterprise-agent trajectories.
  • SciCode & SWE-bench Verified: Used as secondary benchmarks for scientific coding and software engineering tasks.
  • Conditional Memory Utility Dataset: A new, publicly released dataset containing over 16,000 evaluation records with paired outcomes and costs.

Key Insights and Ideas

  • Utility is Sparse: Most memories are "structural zeros" for most tasks. In LongMemEval-V2, the median memory bears on exactly one specific task. Indiscriminate pairing leads to an "absence of evidence" rather than evidence of uselessness.
  • The Compression Tax: Summarization is a double-edged sword. While it saves tokens, it often removes the specific "answer-bearing" strings. The authors found that even when total context length is held fixed, raw text outperforms summaries because it preserves the "needles" in the haystack.
  • "Lost in the Middle" is Real: The utility of a memory dips significantly when it is placed in the middle of the context window compared to the beginning or end.
  • Retrieval is the Bottleneck: Standard retrieval methods (BM25 or dense embeddings) only recover about 40% of answer-bearing memories in their top 10 results. This suggests that "finding the right memory" remains a major open challenge for long-term agents.
Sparsity of utility: most memories bear on very few tasks, and the distribution is heavily skewed toward a single task.

Key Results

  • Headline Contrast: On answer-bearing tasks, adding the memory increased reward by +0.33 compared to matched controls.
  • Raw vs. Compressed: Raw text consistently beat Summary and Structured representations. For example, Raw gained +0.35 at a fixed total context, while Summary gained only +0.04.
  • Robustness: The positive utility of answer-bearing memories held steady across different reader models (GPT-4 vs. Gemma) and different sessions, even though individual "helped/hurt" labels for specific pairs could be unstable.
  • Domain Limits: Interestingly, memories showed no measurable effect on the coding benchmarks (SciCode/SWE-bench), suggesting that factual recall utility doesn't automatically translate to complex reasoning or engineering tasks.
Utility tracks the answer: as more tokens are kept from the raw memory, the likelihood of retaining the answer increases, directly correlating with higher utility ($\Delta U$).

Limitations and Future Work

  • Single-Call Focus: The study focuses on "single-call factual recall." It doesn't account for "procedural memory" (learning how to act) or agents that take multiple steps in an environment.
  • No Multi-Item Interaction: The protocol inserts one memory at a time. It cannot yet measure redundancy (two memories saying the same thing) or complementarity (two memories that only work when combined).
  • Retrieval Gap: The low recall of current retrievers against the "answer-bearing" ground truth highlights a need for better task-aware retrieval mechanisms.
  • Stability: While aggregate trends are robust, individual memory-task labels are sensitive to prompt positioning and model versioning, making it difficult to train "memory gates" on individual data points.

Summary updated

Abstract

A stored experience may answer one future question and offer little help with another. We measure this task-conditioned utility rather than assigning each memory a single importance score. A task-blind writer stores a memory without seeing the later task; a frozen reader then answers that task with and without the memory, using identical filler. A cost ledger records the resources used to form, store and expose the memory. On single-call factual-recall tasks built from LongMemEval-V2 web-agent memories, the same memory helps more when it contains information needed for the answer. Its mean paired reward gain on these answer-bearing task pairs exceeds that on matched-control pairs by $+0.33$ (95% interval $+0.21$ to $+0.46$). The tested task-blind summaries and structured records yield smaller gains than Raw, including at fixed total context; a budget sweep is consistent with retention of answer-bearing content contributing to the difference. The bearing--control contrast remains positive in a new session with randomized memory position and under a second reader, although individual memory--task labels change. Generic retrieval performs above chance but misses most screen-defined answer-bearing memories. The factual-recall gains do not extend to the tested software-engineering and scientific-coding benchmarks, where even raw memories have no measurable effect. We provide a public, versioned Conditional Memory Utility Dataset of paired measurements and costs. The practical takeaway is to evaluate memory compression by the task utility it preserves alongside its costs. Task-conditioned measurements make this comparison possible; their stability must also be checked.
BibTeX
@misc{yang2026what,
  author = {Chih-Hsuan Yang and Anjir Ahmed Chowdhury and Feng Yan and Rajeev Thakur},
  title = {{What Is This Memory Useful For? Measuring Task-Conditioned Utility in Long-Term Agent Memory}},
  year = {2026},
  eprint = {2611.03930},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.03930},
}
Figure