TL;DR
This paper argues that a memory's value isn't a single "importance score" but depends entirely on the specific future task it's asked to help with. By introducing a "task-conditioned" evaluation protocol, the authors demonstrate that common memory compression techniques (like summarization) often "summarize away" the very details needed for factual recall, making raw text surprisingly hard to beat.
What Problem Does This Paper Address?
As LLM agents tackle longer horizons, they face a "memory management" problem: out of thousands of past experiences, which should they keep (retention), which should they look for (retrieval), and which should they actually show the model (exposure)?
Current systems often assign a single importance score to a memory based on recency or a generic "saliency" check. However, a memory of a budget spreadsheet is incredibly useful for a question about finances but useless for a question about software architecture. The authors tackle the challenge of measuring this task-conditioned utility—the specific delta a memory provides for a specific task—while keeping the memory "writer" blind to what those future tasks might be.
Methods
The authors propose a Conditional Memory Utility Protocol designed to isolate the effect of a single memory unit.
- Task-Blind Writing: A "writer" model processes a past experience (e.g., a page visit) into a representation (Raw, Summary, or Structured) without knowing any future questions.
- Counterfactual Evaluation: A "frozen reader" model answers a task twice: once with the target memory and once without it. Crucially, both versions include the same "filler" context to ensure the model isn't just reacting to a longer prompt.
- The Metric ($\Delta U$): Utility is measured as the change in reward (accuracy) between the "with-memory" and "without-memory" arms.
- Answer-Bearing Screen: To avoid drowning in "noise" (since most memories are irrelevant to most tasks), they use a screening tool to identify "answer-bearing" pairs—cases where the memory actually contains the information needed for the gold answer.
Datasets
- LongMemEval-V2: The primary benchmark, consisting of web-agent and enterprise-agent trajectories.
- SciCode & SWE-bench Verified: Used as secondary benchmarks for scientific coding and software engineering tasks.
- Conditional Memory Utility Dataset: A new, publicly released dataset containing over 16,000 evaluation records with paired outcomes and costs.
Key Insights and Ideas
- Utility is Sparse: Most memories are "structural zeros" for most tasks. In LongMemEval-V2, the median memory bears on exactly one specific task. Indiscriminate pairing leads to an "absence of evidence" rather than evidence of uselessness.
- The Compression Tax: Summarization is a double-edged sword. While it saves tokens, it often removes the specific "answer-bearing" strings. The authors found that even when total context length is held fixed, raw text outperforms summaries because it preserves the "needles" in the haystack.
- "Lost in the Middle" is Real: The utility of a memory dips significantly when it is placed in the middle of the context window compared to the beginning or end.
- Retrieval is the Bottleneck: Standard retrieval methods (BM25 or dense embeddings) only recover about 40% of answer-bearing memories in their top 10 results. This suggests that "finding the right memory" remains a major open challenge for long-term agents.
Key Results
- Headline Contrast: On answer-bearing tasks, adding the memory increased reward by +0.33 compared to matched controls.
- Raw vs. Compressed: Raw text consistently beat Summary and Structured representations. For example, Raw gained +0.35 at a fixed total context, while Summary gained only +0.04.
- Robustness: The positive utility of answer-bearing memories held steady across different reader models (GPT-4 vs. Gemma) and different sessions, even though individual "helped/hurt" labels for specific pairs could be unstable.
- Domain Limits: Interestingly, memories showed no measurable effect on the coding benchmarks (SciCode/SWE-bench), suggesting that factual recall utility doesn't automatically translate to complex reasoning or engineering tasks.
Limitations and Future Work
- Single-Call Focus: The study focuses on "single-call factual recall." It doesn't account for "procedural memory" (learning how to act) or agents that take multiple steps in an environment.
- No Multi-Item Interaction: The protocol inserts one memory at a time. It cannot yet measure redundancy (two memories saying the same thing) or complementarity (two memories that only work when combined).
- Retrieval Gap: The low recall of current retrievers against the "answer-bearing" ground truth highlights a need for better task-aware retrieval mechanisms.
- Stability: While aggregate trends are robust, individual memory-task labels are sensitive to prompt positioning and model versioning, making it difficult to train "memory gates" on individual data points.
Summary updated