Diagnosing Retrieval vs. Utilization Bottlenecks in LLM Agent Memory

Boqin Yuan, Yue Su, Kun Yao

TL;DR

Researchers have found that for LLM agents with long-term memory, how you find information is far more important than how you store it. By testing various memory strategies, the study reveals that expensive LLM-driven summarization often performs worse than simply storing raw text chunks, primarily because the real bottleneck lies in retrieval precision rather than the model's ability to understand the context.

What Problem Does This Paper Address?

As we move toward LLM agents that can remember weeks or months of interaction, a design divide has emerged. Some systems store every word (raw chunks), some extract "atomic facts" (like Mem0), and others condense conversations into narrative summaries (like MemGPT).

Until now, it hasn't been clear which approach is best, or why one might fail. If an agent gives a wrong answer, is it because the memory was stored poorly (a "write" error), because the search engine couldn't find the right snippet (a "retrieval" error), or because the LLM ignored the information it was given (a "utilization" error)? This paper introduces a diagnostic framework to pinpoint exactly where the memory pipeline breaks down.

Methods

The authors conducted a 3×3 factorial study, crossing three "Write Strategies" with three "Retrieval Methods":

Write Strategies: 1. Basic RAG: Storing raw 3-turn conversation chunks (lossless, zero LLM cost). 2. Extracted Facts: Using an LLM to pull out self-contained facts and resolve conflicts (inspired by Mem0). 3. Summarized Episodes: Condensing entire sessions into single paragraphs (inspired by MemGPT).

Retrieval Methods: 1. Cosine Similarity: Standard embedding-based semantic search. 2. BM25: Classic keyword-based matching. 3. Hybrid + Rerank: Combining both methods and using a "judge" LLM to pick the best top-k results.

To analyze the results, they used three Diagnostic Probes:

  • Probe 1 (Relevance): Does the retrieved memory actually contain the answer?
  • Probe 2 (Utilization): Does the model's answer improve when it's given the memory vs. when it isn't?
  • Probe 3 (Failure Mode): Categorizing errors into Retrieval Failure, Utilization Failure, or Hallucination.

Datasets

The study used LoCoMo, a long-context conversational memory benchmark. It consists of 10 multi-session conversations (about 600 turns each) and 1,540 non-adversarial questions that require the agent to remember specific details across different sessions.

Key Insights and Ideas

  • Retrieval is the Kingmaker: The choice of retrieval method caused a 20-point swing in accuracy, while the storage strategy only moved the needle by 3–8 points. If you want a better agent, upgrade your retriever, not your summarizer.
  • The "Lossy Compression" Trap: Sophisticated write strategies like fact extraction and summarization are "lossy." They often discard nuances that the LLM could have used to answer a question. Consequently, the "cheapest" method—storing raw chunks—frequently outperformed the expensive LLM-processed alternatives.
  • LLMs are Great Users, Poor Searchers: The failure analysis showed that when the correct memory was successfully retrieved, the LLM almost always used it correctly (utilization failure was rare). The system almost always failed because the retriever simply didn't surface the right information.
  • Semantic vs. Lexical: BM25 (keyword search) performed poorly on its own but was a vital component of the Hybrid approach, helping to find specific names or terms that embeddings might miss.

Key Results

  • Top Performance: The combination of Basic RAG (raw chunks) + Hybrid Reranking achieved the highest accuracy at 81.1%.
  • The Gap: Switching from BM25 to Hybrid Reranking improved average accuracy from 57.1% to 77.2%.
  • Cost-Efficiency: Basic RAG requires zero LLM calls at "write time," making it significantly cheaper and more effective than Summarized Episodes (which requires an LLM call for every session).
  • Correlation: Retrieval precision was nearly perfectly correlated with downstream accuracy ($r=0.98$), confirming that the retrieval stage is the primary bottleneck.

Limitations and Future Work

The authors acknowledge several constraints:

  • Single Model: The study primarily used GPT-5-mini as the backbone; results might vary with smaller or less capable models.
  • Fixed Budget: They limited retrieval to $k=5$ entries. In scenarios with much tighter context windows, the "compression" offered by summarization might become more valuable.
  • Prompt-based vs. Learned: The strategies tested were prompt-based implementations. Future work could explore "learned" memory systems that optimize the writing and retrieval process jointly via reinforcement learning.
  • Judge Reliability: The framework relies on LLM-as-a-judge for diagnostics, which, while validated against human labels (92% agreement), still introduces a layer of model bias.

Summary updated

Abstract

Memory-augmented LLM agents store and retrieve information from prior interactions, yet the relative importance of how memories are written versus how they are retrieved remains unclear. We introduce a diagnostic framework that analyzes how performance differences manifest across write strategies, retrieval methods, and memory utilization behavior, and apply it to a 3x3 study crossing three write strategies (raw chunks, Mem0-style fact extraction, MemGPT-style summarization) with three retrieval methods (cosine, BM25, hybrid reranking). On LoCoMo, retrieval method is the dominant factor: average accuracy spans 20 points across retrieval methods (57.1% to 77.2%) but only 3-8 points across write strategies. Raw chunked storage, which requires zero LLM calls, matches or outperforms expensive lossy alternatives, suggesting that current memory pipelines may discard useful context that downstream retrieval mechanisms fail to compensate for. Failure analysis shows that performance breakdowns most often manifest at the retrieval stage rather than at utilization. We argue that, under current retrieval practices, improving retrieval quality yields larger gains than increasing write-time sophistication. Code is publicly available at https://github.com/boqiny/memory-probe.
BibTeX
@misc{yuan2026diagnosing,
  author = {Boqin Yuan and Yue Su and Kun Yao},
  title = {{Diagnosing Retrieval vs. Utilization Bottlenecks in LLM Agent Memory}},
  year = {2026},
  eprint = {2603.02473},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.02473},
}
Figure