TL;DR
This paper introduces RealMem, a novel benchmark designed to evaluate Large Language Models (LLMs) in realistic, long-term project-oriented interactions. It addresses the limitations of existing benchmarks by simulating complex scenarios where agents must dynamically track evolving goals and maintain coherent memory across multiple sessions, revealing significant challenges for current LLM memory systems.
What Problem Does This Paper Address?
As LLMs transition from simple chatbots to more autonomous agents capable of sustained collaboration, robust memory systems are crucial for maintaining consistency and coherence over extended periods. However, current benchmarks for evaluating LLM memory primarily focus on casual conversations or single-task dialogues. These benchmarks fail to capture the complexities of "long-term project-oriented" interactions, where agents need to manage evolving goals, interleaved user queries across fragmented sessions, and dynamic changes in information states. This gap hinders the development of AI agents that can effectively support real-world, multi-faceted projects.
Methods
The authors propose a three-stage synthesis pipeline to generate realistic, long-term project-oriented dialogues for their benchmark.
- Project Foundation Construction: This stage initializes the static context by defining user personas (demographics, goals) and creating a hierarchical "Project Skeleton." This skeleton outlines high-level milestones (Blueprints), causal dependencies (Events), and guides for individual dialogue sessions (Session Summaries). This ensures global coherence and mitigates fragmentation.
- Multi-Agent Dialogue Generation: A dual-agent framework (User Agent and Assistant Agent) simulates realistic interactions. The User Agent provides session-relevant inputs, while the Assistant Agent receives all relevant memory points and a global schedule to inform its decisions. This stage is structured to simulate concurrent project management by interleaving session summaries.
- Memory and Schedule Management: This stage creates a closed feedback loop by processing generated dialogues. Specialized agents extract salient facts into structured memory points, detect time-sensitive tasks to update a global schedule, and perform semantic-level deduplication to optimize memory storage. This refined memory is then used as context for subsequent dialogue generation.
Datasets
The paper introduces RealMem, a new benchmark comprising over 2,000 cross-session dialogues across eleven realistic project scenarios. These scenarios cover diverse domains like "Life Planning & Management," "Professional & Skill Development," and "Personal Well-being." The dialogues are generated using a sophisticated pipeline that simulates dynamic memory evolution and are evaluated using natural user queries.
Key Insights and Ideas
- The "Project-Oriented" Gap: Existing memory benchmarks are insufficient for evaluating LLMs in real-world scenarios where agents must manage complex, evolving projects over long durations.
- Dynamic Memory is Key: Real-world interactions demand memory systems that can dynamically update, align with evolving contexts, and proactively anticipate user needs, rather than just recalling static facts.
- Hierarchical Structure Matters: A structured approach to building project foundations (personas, blueprints, events) is crucial for maintaining global coherence in long-term dialogues.
- LLM Limitations Exposed: Even state-of-the-art LLMs struggle with the rigorous dependency tracking and strict logical consistency required for complex, technical tasks, highlighting a significant gap in their current memory capabilities.
- Precision Over Recall: For complex project dialogues, high precision (NDCG) in memory retrieval is more critical than high recall, as noisy or irrelevant information can hinder performance.
Key Results
- Benchmark Performance: Existing memory systems, including MemoryOS and Graph Memory, show significant performance variations across different memory categories (e.g., Temporal Reasoning vs. Dynamic Updating) and session types (Single vs. Multi Session).
- MemoryOS Superiority: MemoryOS demonstrates robust performance, particularly in handling dynamic information and proactive alignment, achieving the highest QA scores in several categories.
- Graph Memory's Strength: Graph Memory excels in Temporal Reasoning, suggesting its graph-based structure is effective for tasks involving intricate entity relationships.
- LLM Foundation Models: The study highlights that advanced LLMs (like GPT-40) are necessary for effectively tackling complex queries in RealMem, as even with Oracle memory, smaller models show limited performance.
- Efficiency Trade-offs: Memory incorporation latency is a critical bottleneck. While MemoryOS offers competitive retrieval, it incurs higher token costs, indicating a trade-off between performance and resource usage.
Limitations and Future Work
- Data Generation Reliance: The data construction heavily relies on Gemini 2.5 models, which, while powerful, may raise concerns about reproducibility and cost. GPT series models showed inferior format adherence.
- Scope of Benchmark: RealMem currently focuses specifically on memory-centric challenges and does not yet assess tool use capabilities.
- Future Work: The authors plan to incorporate tool use capabilities into the benchmark to extend its applicability to more complex task processing. They also aim to further investigate memory ingestion mechanisms for scalable deployment and explore more robust memory systems for real-world autonomous agents.
Summary updated