EgoMem: Lifelong Memory Agent for Full-duplex Omnimodal Models

Yiqun Yao, Naitong Yu, Xiang Li, Xin Jiang, Xuezhi Fang, Wenjia Ma, Xuying Meng, Jing Li, Aixin Sun, Yequan WangSubmission to ICLR 2026 · Rejected

TL;DR

EgoMem is the first lifelong memory architecture designed for full-duplex, omnimodal models (such as conversational home robots) that process continuous, real-time audiovisual streams. By decoupling memory operations into three asynchronous processes—biometric polling/retrieval, streaming dialog generation, and background memory management—it allows models to recognize users, track evolving social graphs, and deliver personalized responses without needing pre-segmented text turns or manual logins.

What Problem Does This Paper Address?

Traditional memory agents (e.g., Mem0, MemoryBank) rely on assumptions built around text-based Large Language Models (LLMs): users log in explicitly, interactions occur in clearly demarcated turns and sessions, and all inputs are text.

In lifelong embodied scenarios (such as an embodied assistant or companion robot), these assumptions break down:

  1. Implicit identity & lack of session boundaries: Audio and visual inputs run in an uninterrupted, real-time stream. The agent must continuously infer who is speaking, when a conversation begins or ends, and who changed.
  2. Context window limits: Even multi-million-token context windows cannot handle unbounded, lifelong multi-channel audio/video streams.
  3. Multi-user social structures: Real-world environments contain interconnected networks of people (e.g., family members, colleagues), which standard profile-based memory models ignore.

Methods

EgoMem augments RoboEgo—a native full-duplex speech-and-language model—with a three-process asynchronous memory system operating at two capability tiers: Level-1 (individual user profiles) and Level-2 (content-driven multi-user social graphs).

   ┌─────────────────────── Lifelong Audiovisual Stream ────────────────────────┐
   │                                                                            │
   ▼                                    ▼                                       ▼
[1. Retrieval Process]       [2. Omnimodal Dialog]            [3. Memory Management]
 • 2s periodic polling        • RoboEgo Backbone (7B+100M)     • Episodic Trigger (100M)
 • Face: RetinaFace/FaceNet   • Consumes MemChunk KV-cache     • Sequence labeling (0/1/2/3)
 • Voice: WavLM + S-Norm      • Real-time speech response      • ASR + External LLM extractor
 • Graph: BM25 + BGE-small    • Generates active text queries  • Memory Update Agent resolves conflicts
  1. Retrieval Process (Active Identification):
  2. Runs a 2-second polling loop over incoming audio/visual chunks ($\tau$).
  3. Uses RetinaFace + Facenet512 for face verification and a fine-tuned WavLM-Large model with adaptive s-norm for speaker verification.
  4. Populates a dedicated Level-1 MemChunk (KV-cache tokens) when user presence changes.
  5. For Level-2, when the dialog model actively outputs retrieval query tokens (e.g., <retr>: relation, keywords), EgoMem performs hybrid document retrieval (BM25 for relations + BGE-small dense re-ranking) over connected users' memory entries and populates Level-2 MemChunk.
  6. Omnimodal Dialog Process:
  7. RoboEgo consumes 17-channel input embeddings (listening/speaking audio via the Mimi neural audio codec at 12.5 fps, plus textual monologues) along with the prepended MemChunks.
  8. Generates dual real-time outputs: immediate audio responses and internal textual monologue tokens.
  9. Memory Management Process (Asynchronous Lifecycle):
  10. Episodic Trigger: A 100M-parameter bidirectional RQ-Transformer scans historical audio token windows (8,192 steps / ~11 mins) and tags frames (0: no dialog, 1: start, 2: in-session, 3: end) to delineate user-specific interaction sessions.
  11. Memory Extractor & Update Agent: Clipped session audio is transcribed via ASR; an external LLM (DeepSeek-V3) evaluates the transcript alongside the model's internal monologue to extract facts, dialog summaries, and a 90-dimensional persona trait vector, updating or resolving conflicts in the long-term knowledge base.

Datasets

  • Biometrics: Evaluated on Labeled Faces in the Wild (LFW) for face verification, and VoxCeleb (with imposter cohorts from SeedTTS-eval) for speaker verification.
  • Training Data: Synthesized dialog transcripts generated by prompting DeepSeek-V3 and Gemini-2.5-Pro on top of public instruction-following datasets (Infinity-Instruct, WizardLM) and VQA sets.
  • Utterances were rendered to speech using Fishaudio TTS, augmented with real-world acoustic noise (DNS Challenge, RNNoise) and simulated room echo.
  • 158K samples for Level-1; 54K samples for Level-2.
  • Evaluation Benchmarks: A held-out set of 1,000 multi-turn token streams for the Episodic Trigger; 200 graph-query pairs for text retrieval; and 50 human-annotated turns to calibrate LLM-as-a-judge metrics.

Key Insights and Ideas

  • Unified Token-Stream Supervision: By structuring user profiles, retrieval queries, and dialog turns into a single unified 8,192-step sequence, the authors train boundary detection (Episodic Trigger), query generation, and response generation simply by swapping supervision masks.
  • Self-Correcting Memory Extraction: Relying solely on raw ASR yields brittle memory writes when audio is noisy. By prompting the extraction LLM with both the raw user ASR and the model’s internal monologue (which succeeded in understanding the user in real time), the system effectively repairs transcription errors.
  • Decoupled Identity vs. Reasoning Latency: Background biometric polling means user context is loaded before the user speaks. Even with Level-2 retrieval, total pipeline generation stays above 20 fps—comfortably surpassing the 12.5 fps hardware requirement for fluid conversation.

Key Results

  • Retrieval Components: Achieves 98.4% accuracy on face verification (0.2s), 96.5% pass@1 with an Equal Error Rate (EER) of 0.89% on speaker verification (0.1s), and 96.0% pass@5 on textual graph retrieval (<0.1s).
  • Boundary Segmentation: The Episodic Trigger attains an F1@5 score of 0.986 on clean streams and 0.982 on noisy streams (detecting boundaries within 0.4 seconds).
  • Personalized Dialog: On fact-consistency benchmarks:
  • Level-1: Fact Score of 95.9% (clean) and 93.1% (noisy).
  • Level-2: Fact Score of 89.5% (clean) and 87.6% (noisy).
  • The model maintains an overall throughput of 20.56–21.73 fps during real-time interaction.
  • Human Alignment: LLM evaluators achieved high agreement with human experts (Cohen's Kappa of 0.683 on Fact Score; Pearson $r = 0.621$ on quality).

Limitations and Future Work

  • Single-Speaker Constraint: The framework assumes one active speaker per turn and leaves multi-speaker overlapping speech (the cocktail party problem) for future work.
  • Error Attribution: Error analysis of failure cases indicates that 68% of errors originate from acoustic understanding/instruction parsing, while 22% stem from retrieval failures and 10% from context integration.
  • Modular vs. End-to-End: The architecture relies on external pipeline components (ASR, DeepSeek-V3 API, separate embedding encoders). Future work involves unifying these processes into end-to-end trainable weights and expanding memory formats to include procedural and direct multimodal artifacts.

Summary updated

Abstract

We introduce EgoMem, the first lifelong memory agent tailored for full-duplex models that process real-time omnimodal streams. EgoMem enables real-time models to recognize multiple users directly from raw audiovisual streams, to provide personalized response, and to maintain long-term knowledge of users’ facts, preferences, and social relationships extracted from audiovisual history. EgoMem operates with three asynchronous processes: (i) a retrieval process that dynamically identifies user via face and voice, and gathers relevant context from a long-term memory; (ii) an omnimodal dialog process that generates personalized audio responses based on the retrieved context; and (iii) a memory management process that automatically detects dialog boundaries from omnimodal streams, and extracts necessary information to update the long-term memory. Unlike existing memory agents for LLMs, EgoMem relies entirely on raw audiovisual streams, making it especially suitable for lifelong, real-time, and embodied scenarios. Experimental results demonstrate that EgoMem’s retrieval and memory management modules achieve over 95% accuracy on the test set. When integrated with a fine-tuned RoboEgo omnimodal chatbot, the system achieves fact-consistency scores above 87% in real-time personalized dialogs, establishing a strong baseline for future research.
BibTeX
@misc{yao2025egomem,
  author = {Yiqun Yao and Naitong Yu and Xiang Li and Xin Jiang and Xuezhi Fang and Wenjia Ma and Xuying Meng and Jing Li and Aixin Sun and Yequan Wang},
  title = {{EgoMem: Lifelong Memory Agent for Full-duplex Omnimodal Models}},
  year = {2025},
  eprint = {2509.11914},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.11914},
}
Figure