RealMem: Benchmarking LLMs in Real-World Memory-Driven Interaction

Haonan Bian, Zhiyuan Yao, Sen Hu, Zishan Xu, Shaolei Zhang, Yifu Guo, Ziliang Yang, Xueran Han, Huacan Wang, Ronghao Chen

TL;DR

This paper introduces RealMem, a novel benchmark designed to evaluate Large Language Models (LLMs) in realistic, long-term project-oriented interactions. It addresses the limitations of existing benchmarks by simulating complex scenarios where agents must dynamically track evolving goals and maintain coherent memory across multiple sessions, revealing significant challenges for current LLM memory systems.

What Problem Does This Paper Address?

As LLMs transition from simple chatbots to more autonomous agents capable of sustained collaboration, robust memory systems are crucial for maintaining consistency and coherence over extended periods. However, current benchmarks for evaluating LLM memory primarily focus on casual conversations or single-task dialogues. These benchmarks fail to capture the complexities of "long-term project-oriented" interactions, where agents need to manage evolving goals, interleaved user queries across fragmented sessions, and dynamic changes in information states. This gap hinders the development of AI agents that can effectively support real-world, multi-faceted projects.

Methods

The authors propose a three-stage synthesis pipeline to generate realistic, long-term project-oriented dialogues for their benchmark.

  1. Project Foundation Construction: This stage initializes the static context by defining user personas (demographics, goals) and creating a hierarchical "Project Skeleton." This skeleton outlines high-level milestones (Blueprints), causal dependencies (Events), and guides for individual dialogue sessions (Session Summaries). This ensures global coherence and mitigates fragmentation.
  2. Multi-Agent Dialogue Generation: A dual-agent framework (User Agent and Assistant Agent) simulates realistic interactions. The User Agent provides session-relevant inputs, while the Assistant Agent receives all relevant memory points and a global schedule to inform its decisions. This stage is structured to simulate concurrent project management by interleaving session summaries.
  3. Memory and Schedule Management: This stage creates a closed feedback loop by processing generated dialogues. Specialized agents extract salient facts into structured memory points, detect time-sensitive tasks to update a global schedule, and perform semantic-level deduplication to optimize memory storage. This refined memory is then used as context for subsequent dialogue generation.

Datasets

The paper introduces RealMem, a new benchmark comprising over 2,000 cross-session dialogues across eleven realistic project scenarios. These scenarios cover diverse domains like "Life Planning & Management," "Professional & Skill Development," and "Personal Well-being." The dialogues are generated using a sophisticated pipeline that simulates dynamic memory evolution and are evaluated using natural user queries.

Key Insights and Ideas

  • The "Project-Oriented" Gap: Existing memory benchmarks are insufficient for evaluating LLMs in real-world scenarios where agents must manage complex, evolving projects over long durations.
  • Dynamic Memory is Key: Real-world interactions demand memory systems that can dynamically update, align with evolving contexts, and proactively anticipate user needs, rather than just recalling static facts.
  • Hierarchical Structure Matters: A structured approach to building project foundations (personas, blueprints, events) is crucial for maintaining global coherence in long-term dialogues.
  • LLM Limitations Exposed: Even state-of-the-art LLMs struggle with the rigorous dependency tracking and strict logical consistency required for complex, technical tasks, highlighting a significant gap in their current memory capabilities.
  • Precision Over Recall: For complex project dialogues, high precision (NDCG) in memory retrieval is more critical than high recall, as noisy or irrelevant information can hinder performance.

Key Results

  • Benchmark Performance: Existing memory systems, including MemoryOS and Graph Memory, show significant performance variations across different memory categories (e.g., Temporal Reasoning vs. Dynamic Updating) and session types (Single vs. Multi Session).
  • MemoryOS Superiority: MemoryOS demonstrates robust performance, particularly in handling dynamic information and proactive alignment, achieving the highest QA scores in several categories.
  • Graph Memory's Strength: Graph Memory excels in Temporal Reasoning, suggesting its graph-based structure is effective for tasks involving intricate entity relationships.
  • LLM Foundation Models: The study highlights that advanced LLMs (like GPT-40) are necessary for effectively tackling complex queries in RealMem, as even with Oracle memory, smaller models show limited performance.
  • Efficiency Trade-offs: Memory incorporation latency is a critical bottleneck. While MemoryOS offers competitive retrieval, it incurs higher token costs, indicating a trade-off between performance and resource usage.

Limitations and Future Work

  • Data Generation Reliance: The data construction heavily relies on Gemini 2.5 models, which, while powerful, may raise concerns about reproducibility and cost. GPT series models showed inferior format adherence.
  • Scope of Benchmark: RealMem currently focuses specifically on memory-centric challenges and does not yet assess tool use capabilities.
  • Future Work: The authors plan to incorporate tool use capabilities into the benchmark to extend its applicability to more complex task processing. They also aim to further investigate memory ingestion mechanisms for scalable deployment and explore more robust memory systems for real-world autonomous agents.

Summary updated

Abstract

As Large Language Models (LLMs) evolve from static dialogue interfaces to autonomous general agents, effective memory is paramount to ensuring long-term consistency. However, existing benchmarks primarily focus on casual conversation or task-oriented dialogue, failing to capture **"long-term project-oriented"** interactions where agents must track evolving goals. To bridge this gap, we introduce **RealMem**, the first benchmark grounded in realistic project scenarios. RealMem comprises over 2,000 cross-session dialogues across eleven scenarios, utilizing natural user queries for evaluation. We propose a synthesis pipeline that integrates Project Foundation Construction, Multi-Agent Dialogue Generation, and Memory and Schedule Management to simulate the dynamic evolution of memory. Experiments reveal that current memory systems face significant challenges in managing the long-term project states and dynamic context dependencies inherent in real-world projects. Our code and datasets are available at [https://github.com/AvatarMemory/RealMemBench](https://github.com/AvatarMemory/RealMemBench).
BibTeX
@inproceedings{bian2026realmem,
  author = {Haonan Bian and Zhiyuan Yao and Sen Hu and Zishan Xu and Shaolei Zhang and Yifu Guo and Ziliang Yang and Xueran Han and Huacan Wang and Ronghao Chen},
  title = {{RealMem: Benchmarking LLMs in Real-World Memory-Driven Interaction}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.703},
}

Loading related content...

Figure