MemBubble: Benchmarking How Agent Memory Shapes LLM Response Diversity

Jiajun Tan, Fei Sun, Xinyu Ma, Shixuan Zhang, Du Su, Qi Cao, Daiting Shi, Huawei Shen, Xueqi ChengarXiv 2026 · arXiv

TL;DR

MemBubble is a new benchmark designed to reveal how persistent memory in LLM agents creates "filter bubbles," significantly narrowing the range of perspectives and information provided to users. The study finds that across various LLMs and memory architectures, response diversity (measured by entropy over functional equivalence classes) drops by an average of 34.7%, a gap that simple fixes like increasing sampling temperature or prompt engineering fail to close.

What Problem Does This Paper Address?

As LLM agents move from simple chatbots to personalized assistants, they increasingly rely on persistent memory to track user preferences over time. While this enables helpful personalization, it introduces a significant risk: diversity loss.

If an agent "remembers" your biases or specific tastes, it may stop presenting alternative viewpoints or diverse options, even when the current query is open-ended and doesn't explicitly call for a personalized answer. This is the AI equivalent of the "filter bubble" seen in social media algorithms. Most existing evaluations of this effect rely on synthetic, "clean" dialogues. This paper addresses the lack of a real-world, systematic benchmark to measure how actual memory systems (like RAG or summarization-based architectures) impact the variety of an agent's responses.

Figure 1: Illustration of response diversity decrease as memory steers responses toward similar preference- aligned content, potentially reinforcing existing biases.

Methods

The authors developed a rigorous pipeline to construct the MemBubble benchmark:

  1. Preference Extraction: They filtered real-world user-assistant dialogues to find stable user preferences (e.g., "The user thinks Christopher Nolan's films are great").
  2. Scenario Shifting: To ensure they weren't just measuring "correct" personalization, they generated test queries in a different scenario but within the same category. For example, if the preference is about a director for a movie, the test query might ask for a guest speaker at a comedy workshop.
  3. Functional Equivalence: Instead of using simple word-overlap metrics, they defined diversity based on "functional equivalence." Two responses are equivalent if they provide the same utility to the user (e.g., recommending the same person using different words). They used an LLM judge to partition 10 sampled responses into these equivalence classes.
  4. Memory Architectures: They tested five representative memory systems: MemoryBank, A-Mem, mem0, MemoryOS, and NaiveRAG, comparing them against "No Memory" and "Full History" baselines.
Figure 2: Construction of MemBubble benchmark. Each instance pairs a multi-session interaction history assembled from a preference dialogue and sampled background dialogues with an open-ended test query.

Datasets

The study introduces the MemBubble dataset:

  • Size: 1,000 instances.
  • Source: Built from real-world interactions in LMSYS-Chat-1M and WildChat.
  • Diversity: Covers 35 languages and 14 topic categories (STEM, Art & Literature, and Society being the largest).
  • Structure: Each instance includes a multi-session interaction history (up to 30,000 tokens) and an open-ended test query.
Figure 3: Overview of our proposed MemBubble dataset.

Key Insights and Ideas

  • The "Processing Penalty": Surprisingly, specialized memory systems (which summarize or selectively retrieve) cause more diversity loss than simply stuffing the entire interaction history into the prompt. This suggests that the way we currently "compress" or "index" memories is a primary driver of the filter bubble.
  • Personalization vs. Diversity Trade-off: There is a direct correlation between "preference hit rate" (how often the model gives the user what it thinks they want) and diversity loss. The more "personalized" the system feels, the narrower its output becomes.
  • Ineffective Interventions: Common "tricks" to increase LLM creativity—like cranking up the sampling temperature, using system prompts to "be diverse," or rephrasing the query—do not effectively bridge the diversity gap created by memory.
  • Consistency Across Models: The trend was consistent across both closed-source (GPT-6-Sol, Gemini 3.7 Flash) and open-source (Llama 3.3-70B) models, though the magnitude varied.

Key Results

  • Entropy Drop: Across all tested memory mechanisms, class entropy decreased by 34.7% on average compared to providing the full history.
  • Distinct-k Reduction: The number of unique functional classes (distinct-k) dropped by 25.7%.
  • The "One-Track Mind": Under memory-augmented conditions, roughly 34% of samples resulted in a distinct-k of 1 (meaning all 10 samples were functionally identical), compared to only 6% in the "No Memory" baseline.
Figure 4: Distribution of distinct-k of MemBubble samples across memory conditions. Dashed vertical lines indicate the mean for each condition.

Limitations and Future Work

  • Static Evaluation: The benchmark uses fixed interaction histories. It doesn't yet capture the "feedback loop" where an agent's narrow responses might further narrow a user's expressed preferences over months of use.
  • Task Scope: The study focuses on open-ended tasks. The authors acknowledge that for some tasks, high personalization (and thus low diversity) is exactly what the user wants; the challenge is determining when to prioritize which.
  • Mitigation Strategies: While simple interventions failed, the authors leave the development of more sophisticated, diversity-preserving memory retrieval algorithms to future work.

Summary updated

Abstract

Memory serves as the core component of LLM-based agents, enabling personalized responses via extracting and using user preferences from past interactions. However, its use may reduce response diversity, thereby narrowing the range of information available to users. Existing evaluations rely largely on synthetic interactions and cover few memory systems, leaving the effects of memory on response diversity insufficiently evaluated in real-world scenarios. We introduce MemBubble, a benchmark of 1000 instances built on preferences extracted from real user-assistant conversations. To measure the extent to which memory concentrates response distributions, we construct test queries in scenarios distinct from user preferences while preserving topical relevance through shared preference categories. Across multiple LLMs and memory mechanisms, entropy decreases by 34.7% on average compared with directly providing the full interaction history, highlighting the important role of memory processing in diversity loss. We also explore simple user-side interventions and changes in information availability but find that these do not effectively close the diversity gap. Our findings highlight the need to consider preserving diversity and user access to information alongside improvements in personalization when designing and using agent memory systems.
BibTeX
@misc{tan2026membubble,
  author = {Jiajun Tan and Fei Sun and Xinyu Ma and Shixuan Zhang and Du Su and Qi Cao and Daiting Shi and Huawei Shen and Xueqi Cheng},
  title = {{MemBubble: Benchmarking How Agent Memory Shapes LLM Response Diversity}},
  year = {2026},
  eprint = {2610.25127},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.25127},
}
Figure