Abstract
While Large Language Models (LLMs) have demonstrated impressive capabilities in processing long contexts, their ability to maintain "very long-term" conversational memory—spanning months and dozens of sessions—remains largely unexplored. This paper introduces LOCOMO, a first-of-its-kind multimodal dataset designed to evaluate long-range temporal and causal reasoning in dialogue agents. Our evaluation reveals a significant "memory gap": even state-of-the-art models like GPT-4 struggle with long-range temporal and causal reasoning, lagging nearly 55% behind human performance in complex dialogue tasks.
1. Introduction
Large Language Models (LLMs) are increasingly deployed as persistent conversational agents. However, most open-domain dialogue research focuses on "short-term" long-term memory, typically involving contexts under 1,000 tokens across five or fewer sessions. Real-world human relationships involve much more complex dynamics, requiring the ability to remember distant facts and understand how past events causally influence the present. For example, understanding how a job change months ago led to a move and subsequent lifestyle shifts requires sophisticated temporal and causal reasoning. Existing benchmarks fail to capture these multi-session dynamics, particularly in multimodal contexts where images are shared and discussed.
2. Methodology: The LOCOMO Pipeline
To address the lack of high-quality, long-term data, we developed a novel machine-human pipeline to generate consistent, very long-term dialogues. The process consists of four primary stages:
- Persona & Event Grounding: Two LLM agents are initialized with distinct personas and a "Temporal Event Graph." This graph serves as a timeline of causally linked life events (e.g., "enrolling in a course" leads to "excitement on social media"), ensuring the agents have a grounded history to reference.
- Virtual Agent Architecture: Agents utilize a "Reflect & Respond" mechanism. After each session, the agent generates a summary (short-term memory) and extracts "observations" (long-term memory assertions) to guide future interactions and maintain consistency.
- Multimodal Integration: Agents are equipped to share and react to images. These images are sourced via web search based on the specific dialogue context to simulate real-world multimodal interaction.
- Human-in-the-Loop Refinement: To ensure high data quality, human annotators edit the LLM-generated dialogues. This step is crucial for fixing long-range inconsistencies, removing irrelevant images, and ensuring strict adherence to the underlying event graphs.
3. The LOCOMO Dataset
The resulting dataset, LOCOMO (Long-term COnversational Multi-modal), represents a significant leap in scale and complexity over previous benchmarks:
- Scale: The dataset includes 50 conversations, averaging 300 turns and 9,209 tokens per conversation.
- Duration: Conversations span up to 35 sessions, simulating interactions over several months.
- Comparative Depth: LOCOMO is approximately 9x longer in token count and covers 4x more sessions than the previous standard, Multi-Session Chat (MSC).
- Evaluation Tasks: The benchmark includes 7,512 QA pairs covering single-hop, multi-hop, temporal, and adversarial reasoning, alongside event summarization and multimodal generation tasks.
4. Key Insights and Analysis
Our evaluation of current LLMs on the LOCOMO dataset yielded several critical insights into the limitations of current memory architectures:
4.1 The Hallucination Trap
Long-context LLMs (such as GPT-3.5-16k) exhibit a massive performance drop—up to 83% lower than base models—when faced with adversarial questions. In massive contexts, these models become "gullible," often being misled into confirming false premises or misassigning events to the wrong speaker.
4.2 Observations vs. Raw Logs
For Retrieval-Augmented Generation (RAG), we found that providing the model with a database of distilled "observations" (e.g., "Joanna is allergic to cats") is significantly more effective than providing raw conversation logs. This approach improves the signal-to-noise ratio, allowing the model to focus on relevant facts rather than conversational filler.
4.3 The Temporal Wall
Temporal reasoning remains the most significant challenge for all tested models. Tasks such as determining the duration between two distant events (e.g., "How long after the move did Nate get a dog?") proved difficult, as LLMs struggle to map the passage of time across fragmented chat sessions.
4.4 Summarization Fatigue
Interestingly, long-context models did not outperform base models in event summarization tasks. This suggests that a larger context window does not inherently improve a model's ability to synthesize a coherent narrative from a long history.
5. Results
The performance gap between AI and humans remains stark. On the Question Answering task, the best-performing model, GPT-4-turbo, achieved an overall F1 score of 32.1, while the human benchmark reached 87.9. While RAG with "observations" improved GPT-3.5's performance by approximately 5% over raw logs, it still falls far short of human-level consistency. In multimodal generation, incorporating retrieved observations helped models like MiniGPT-5 maintain persona consistency, though performance still decayed as the dialogue history grew.
6. Conclusion and Future Work
LOCOMO highlights the significant challenges remaining in developing agents with true long-term memory. Current models struggle with temporal reasoning and are highly susceptible to hallucinations in long contexts. Future research should focus on improving signal-to-noise filtering in RAG systems and developing architectures capable of maintaining a causal "chain of events" over months of interaction. Furthermore, expanding the dataset to include organic human nuances and more diverse languages will be essential for creating truly robust conversational agents.
Summary updated