TL;DR
EgoMem is the first lifelong memory architecture designed for full-duplex, omnimodal models (such as conversational home robots) that process continuous, real-time audiovisual streams. By decoupling memory operations into three asynchronous processes—biometric polling/retrieval, streaming dialog generation, and background memory management—it allows models to recognize users, track evolving social graphs, and deliver personalized responses without needing pre-segmented text turns or manual logins.
What Problem Does This Paper Address?
Traditional memory agents (e.g., Mem0, MemoryBank) rely on assumptions built around text-based Large Language Models (LLMs): users log in explicitly, interactions occur in clearly demarcated turns and sessions, and all inputs are text.
In lifelong embodied scenarios (such as an embodied assistant or companion robot), these assumptions break down:
- Implicit identity & lack of session boundaries: Audio and visual inputs run in an uninterrupted, real-time stream. The agent must continuously infer who is speaking, when a conversation begins or ends, and who changed.
- Context window limits: Even multi-million-token context windows cannot handle unbounded, lifelong multi-channel audio/video streams.
- Multi-user social structures: Real-world environments contain interconnected networks of people (e.g., family members, colleagues), which standard profile-based memory models ignore.
Methods
EgoMem augments RoboEgo—a native full-duplex speech-and-language model—with a three-process asynchronous memory system operating at two capability tiers: Level-1 (individual user profiles) and Level-2 (content-driven multi-user social graphs).
┌─────────────────────── Lifelong Audiovisual Stream ────────────────────────┐
│ │
▼ ▼ ▼
[1. Retrieval Process] [2. Omnimodal Dialog] [3. Memory Management]
• 2s periodic polling • RoboEgo Backbone (7B+100M) • Episodic Trigger (100M)
• Face: RetinaFace/FaceNet • Consumes MemChunk KV-cache • Sequence labeling (0/1/2/3)
• Voice: WavLM + S-Norm • Real-time speech response • ASR + External LLM extractor
• Graph: BM25 + BGE-small • Generates active text queries • Memory Update Agent resolves conflicts
- Retrieval Process (Active Identification):
- Runs a 2-second polling loop over incoming audio/visual chunks ($\tau$).
- Uses RetinaFace + Facenet512 for face verification and a fine-tuned WavLM-Large model with adaptive s-norm for speaker verification.
- Populates a dedicated
Level-1 MemChunk(KV-cache tokens) when user presence changes. - For Level-2, when the dialog model actively outputs retrieval query tokens (e.g.,
<retr>: relation, keywords), EgoMem performs hybrid document retrieval (BM25 for relations + BGE-small dense re-ranking) over connected users' memory entries and populatesLevel-2 MemChunk. - Omnimodal Dialog Process:
- RoboEgo consumes 17-channel input embeddings (listening/speaking audio via the Mimi neural audio codec at 12.5 fps, plus textual monologues) along with the prepended
MemChunks. - Generates dual real-time outputs: immediate audio responses and internal textual monologue tokens.
- Memory Management Process (Asynchronous Lifecycle):
- Episodic Trigger: A 100M-parameter bidirectional RQ-Transformer scans historical audio token windows (8,192 steps / ~11 mins) and tags frames (
0: no dialog,1: start,2: in-session,3: end) to delineate user-specific interaction sessions. - Memory Extractor & Update Agent: Clipped session audio is transcribed via ASR; an external LLM (DeepSeek-V3) evaluates the transcript alongside the model's internal monologue to extract facts, dialog summaries, and a 90-dimensional persona trait vector, updating or resolving conflicts in the long-term knowledge base.
Datasets
- Biometrics: Evaluated on Labeled Faces in the Wild (LFW) for face verification, and VoxCeleb (with imposter cohorts from SeedTTS-eval) for speaker verification.
- Training Data: Synthesized dialog transcripts generated by prompting DeepSeek-V3 and Gemini-2.5-Pro on top of public instruction-following datasets (Infinity-Instruct, WizardLM) and VQA sets.
- Utterances were rendered to speech using Fishaudio TTS, augmented with real-world acoustic noise (DNS Challenge, RNNoise) and simulated room echo.
- 158K samples for Level-1; 54K samples for Level-2.
- Evaluation Benchmarks: A held-out set of 1,000 multi-turn token streams for the Episodic Trigger; 200 graph-query pairs for text retrieval; and 50 human-annotated turns to calibrate LLM-as-a-judge metrics.
Key Insights and Ideas
- Unified Token-Stream Supervision: By structuring user profiles, retrieval queries, and dialog turns into a single unified 8,192-step sequence, the authors train boundary detection (Episodic Trigger), query generation, and response generation simply by swapping supervision masks.
- Self-Correcting Memory Extraction: Relying solely on raw ASR yields brittle memory writes when audio is noisy. By prompting the extraction LLM with both the raw user ASR and the model’s internal monologue (which succeeded in understanding the user in real time), the system effectively repairs transcription errors.
- Decoupled Identity vs. Reasoning Latency: Background biometric polling means user context is loaded before the user speaks. Even with Level-2 retrieval, total pipeline generation stays above 20 fps—comfortably surpassing the 12.5 fps hardware requirement for fluid conversation.
Key Results
- Retrieval Components: Achieves 98.4% accuracy on face verification (0.2s), 96.5% pass@1 with an Equal Error Rate (EER) of 0.89% on speaker verification (0.1s), and 96.0% pass@5 on textual graph retrieval (<0.1s).
- Boundary Segmentation: The Episodic Trigger attains an F1@5 score of 0.986 on clean streams and 0.982 on noisy streams (detecting boundaries within 0.4 seconds).
- Personalized Dialog: On fact-consistency benchmarks:
- Level-1: Fact Score of 95.9% (clean) and 93.1% (noisy).
- Level-2: Fact Score of 89.5% (clean) and 87.6% (noisy).
- The model maintains an overall throughput of 20.56–21.73 fps during real-time interaction.
- Human Alignment: LLM evaluators achieved high agreement with human experts (Cohen's Kappa of 0.683 on Fact Score; Pearson $r = 0.621$ on quality).
Limitations and Future Work
- Single-Speaker Constraint: The framework assumes one active speaker per turn and leaves multi-speaker overlapping speech (the cocktail party problem) for future work.
- Error Attribution: Error analysis of failure cases indicates that 68% of errors originate from acoustic understanding/instruction parsing, while 22% stem from retrieval failures and 10% from context integration.
- Modular vs. End-to-End: The architecture relies on external pipeline components (ASR, DeepSeek-V3 API, separate embedding encoders). Future work involves unifying these processes into end-to-end trainable weights and expanding memory formats to include procedural and direct multimodal artifacts.
Summary updated