M2A: Multimodal Memory Agent with Dual-Layer Hybrid Memory for Long-Term Personalized Interactions

Junyu Feng, Binxiao Xu, Jiayi Chen, Mengyu Dai, Cenyang Wu, Haodong Li, Bohan Zeng, Yunliu Xie, Hao Liang, Ming Lu, Wentao ZhangarXiv 2026 · arXiv · OpenReview

TL;DR

This paper introduces M²A, a novel multimodal memory agent designed for long-term personalized human-AI interactions. M²A tackles the challenge of evolving user preferences by employing a dual-layer hybrid memory system that continuously updates and refines user information, moving beyond static personalization methods.

What Problem Does This Paper Address?

Current AI systems struggle with long-term personalized interactions, especially when conversation history becomes extensive. Existing personalization methods often treat user preferences as static, failing to adapt to evolving concepts, aliases, and preferences over time. Furthermore, large language models have limited context windows, making it difficult to retain and utilize information from extended conversations. M²A aims to solve this by creating an AI agent that can dynamically learn and adapt to a user's changing profile across weeks or months of interaction, going beyond simple retrieval to a more active memory management system.

Methods

M²A employs a multi-agent architecture with two core components:

  • ChatAgent: This agent handles user interactions, autonomously deciding when to query or update the memory based on a three-stage ReAct-style workflow (Query, Generate, Update).
  • MemoryManager: This agent acts as the backend, managing all read/write operations on the memory bank. It performs iterative, reasoning-driven retrieval and updates, progressively narrowing down information from high-level semantic summaries to specific raw conversational evidence.

The system's novelty lies in its Dual-Layer Hybrid Memory:

  • Raw Message Store: An immutable, append-only log of all raw conversational messages, preserving the complete history.
  • Semantic Memory Store: Stores high-level semantic observations, including textual descriptions, automatically generated image captions, and associated images. Crucially, each semantic entry is linked to its supporting raw messages via evidence_ids.

This dual-layer structure enables progressive narrowing: starting with broad semantic searches and drilling down to specific evidence when needed. M²A also utilizes a tri-path hybrid retrieval strategy that combines dense text embeddings, BM25 sparse retrieval, and cross-modal image embeddings to ensure robust recall across different query types.

Datasets

The authors constructed a new multimodal dataset based on the LoCoMo framework. This dataset involves:

  1. Concept Grouping: Sampling semantically distinct concept groups to simulate realistic discussions.
  2. Unified Multimodal Generation: Using GPT-4 to generate multi-session dialogues and QA pairs, ensuring consistency between visual evidence and narrative.
  3. Temporal Interpolation: Seamlessly integrating generated sessions into original long-context conversations while maintaining temporal coherence.
  4. Hybrid QA Injection: Including reasoning questions (multi-hop, temporal, open-domain) and visual retrieval questions.

This enhanced dataset was used for training and evaluation, with a focus on Visual-Centric questions to specifically test multimodal reasoning.

Key Insights and Ideas

  • Personalization as an Evolving Process: The core idea is that personalization isn't a one-time setup but a continuous, co-evolving process. M²A's ability to incrementally absorb and refine user concepts is key.
  • Dual-Layer Memory for Granularity: The separation of raw logs and semantic summaries, linked by evidence, is crucial for balancing memory efficiency and retrieval precision. This allows for both quick access to general ideas and deep dives into specific details.
  • Agentic Collaboration for Memory Management: Decoupling user interaction (ChatAgent) from memory operations (MemoryManager) simplifies complexity and allows each agent to specialize, leading to more effective memory handling.
  • Tri-Path Retrieval for Multimodal Robustness: Combining different retrieval methods (dense text, sparse text, cross-modal) is essential for handling diverse user queries, especially those involving images.
  • Evidence Linking is Critical: The evidence_ids linking semantic memories to raw messages are vital for verifying and refining semantic information, preventing information loss and enabling correction.

Key Results

M²A consistently outperformed baseline models (LoCoMo, Mem0, A-MEM) across various question categories, including Temporal, Open Domain, Single Hop, and the newly introduced Visual-Centric questions.

  • On GPT-40-mini, M²A achieved an average accuracy of 44.64%, significantly surpassing baselines.
  • For Visual-Centric questions, M²A reached 43.27% accuracy, demonstrating its strength in multimodal reasoning.
  • Ablation studies showed that removing the dual-layer memory degraded performance by 13.31%, iterative retrieval by 16.02%, and the tri-path retrieval by 4.10%, highlighting the importance of each component.
  • Providing even a small amount of recent conversational context (around 5 turns) significantly improved performance, while longer contexts offered diminishing returns.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that while M²A improves personalization, it still relies on LLM-as-a-Judge for evaluation, which has inherent limitations.
  • The performance gap between M²A and some baselines narrowed slightly on the GLM-4.6V-Flash model, suggesting potential for further optimization.
  • Future Work:
    • The authors suggest exploring hybrid approaches that combine editable long-term memory with controllable generation capabilities, potentially leading to even more expressive and personalized AI systems.
  • Further research could focus on more sophisticated methods for handling contradictory information and ensuring the long-term consistency of the memory bank.

Summary updated

Abstract

This work addresses the challenge of personalized question answering in long-term human-machine interactions: when conversational history spans weeks or months and exceeds the context window, existing personalization mechanisms struggle to continuously absorb and leverage users' incremental concepts, aliases, and preferences. Current personalized multimodal models are predominantly static-concepts are fixed at initialization and cannot evolve during interactions. We propose M2A, an agentic dual-layer hybrid memory system that maintains personalized multimodal information through online updates. The system employs two collaborative agents: ChatAgent manages user interactions and autonomously decides when to query or update memory, while MemoryManager breaks down memory requests from ChatAgent into detailed operations on the dual-layer memory bank, which couples a RawMessageStore (immutable conversation log) with a SemanticMemoryStore (high-level observations), providing memories at different granularities. In addition, we develop a reusable data synthesis pipeline that injects concept-grounded sessions from Yo'LLaVA and MC-LLaVA into LoCoMo long conversations while preserving temporal coherence. Experiments show that M2A significantly outperforms baselines, demonstrating that transforming personalization from one-shot configuration to a co-evolving memory mechanism provides a viable path for high-quality individualized responses in long-term multimodal interactions. The code is available at https://github.com/Little-Fridge/M2A.
BibTeX
@misc{feng2026m,
  author = {Junyu Feng and Binxiao Xu and Jiayi Chen and Mengyu Dai and Cenyang Wu and Haodong Li and Bohan Zeng and Yunliu Xie and Hao Liang and Ming Lu and Wentao Zhang},
  title = {{M2A: Multimodal Memory Agent with Dual-Layer Hybrid Memory for Long-Term Personalized Interactions}},
  year = {2026},
  eprint = {2602.07624},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2602.07624},
}
Figure