EgoLife: Towards Egocentric Life Assistant

Jingkang Yang, Shuai Liu, Hongming Guo, Yuhao Dong, Xiamengwei Zhang, Sicheng Zhang, Pengyun Wang, Zitang Zhou, Binzhu Xie, Ziyue Wang, Bei Ouyang, Zhengyu Lin, Marco Cominelli, Zhongang Cai, Yuanhan Zhang, Peiyuan Zhang, Fangzhou Hong, Joerg Widmer, Francesco Gringoli, Lei Yang, Bo Li, Ziwei Liu

TL;DR

The EgoLife project introduces a massive 300-hour, multi-person, multimodal egocentric dataset captured over a full week to pave the way for true AI life assistants. Alongside this dataset, the authors present the EgoLifeQA benchmark for long-context daily reasoning and EgoButler, an AI agent that combines an omni-modal video-audio model (EgoGPT) with a hierarchical retrieval system (EgoRAG) to successfully answer questions spanning days of continuous footage.

What Problem Does This Paper Address?

While egocentric (first-person) vision has advanced rapidly with datasets like Epic-Kitchens and Ego4D, existing resources suffer from two major limitations: they feature short recording durations and focus almost exclusively on a single person's isolated perspective. Real life, however, is continuous and deeply social. Current models cannot track long-term personal habits or understand the complex multi-person dynamics of a shared living space.

To bridge this gap, the authors created a fully instrumented environment where multiple participants lived together for a week, capturing the rich, messy reality of daily life—including multi-person conversations, shared cooking, chores, and socializing.

Figure 1. The Overview of EgoLife Project. The EgoLife project features six participants living together for a week to prepare an Earth Day celebration. Each participant wears Meta Aria glasses [1], recording approximately 8 hours of egocentric video and signals daily. In addition, 15 cameras and 2 mmWave devices provide synchronized third-person perspective data (detailed in Figure 2). These comprehensive annotations enable the development of state-of-the-art multimodal egocentric AI assistants and introduce novel tasks to advance long-term egocentric life assistance, as illustrated in the EgoLife task board.

Methods

The authors developed EgoButler, a dual-system agent designed to handle clip-level understanding and long-horizon retrieval:

  1. EgoGPT (System-I): A lightweight (7B) omni-modal vision-audio-language model built on top of LLaVA-OneVision. To handle the critical audio component of egocentric data, the authors integrated a dedicated audio branch using Whisper Large v3. EgoGPT continuously processes 30-second video clips to generate dense visual-audio captions. Crucially, the model can be personalized by fine-tuning on a participant's first day of data to improve identity-aware tracking.
  2. EgoRAG (System-II): Standard video models cannot ingest a week's worth of video frames at once. EgoRAG solves this by building a hierarchical structured memory bank. Captions generated by EgoGPT are summarized by a language model at hourly and daily levels. When a user asks a question, EgoRAG performs a top-down search (Day $\rightarrow$ Hour $\rightarrow$ Clip) using a scoring function that balances visual and textual relevance to retrieve the top-3 most relevant 30-second clips for EgoGPT to analyze.

Datasets

The paper introduces and utilizes three distinct datasets:

  • EgoLife Dataset: A 300-hour dataset tracking 6 participants living together for 7 days. Each participant wore Meta Aria glasses (recording egocentric video, audio, IMU, and gaze). The environment was augmented with 15 synchronized third-person GoPro cameras and 2 mmWave radar devices for spatial tracking.
  • EgoLifeQA Benchmark: A curated set of 3,000 high-quality, multiple-choice question-answering pairs (500 per participant). The questions require a "look-back" time of at least 5 minutes (and up to multiple days) and are split into five real-world tasks: EntityLog (object tracking/pricing), EventRecall (past events), HabitInsight (personal habits), RelationMap (social interactions), and TaskMaster (task management).
  • EgoIT-99K: A newly curated dataset of 99,480 QA pairs derived from 9 classic egocentric datasets, used to instruction-tune EgoGPT on first-person perspectives before deployment.
Figure 5. Question Types and Examples in the EgoLifeQA Benchmark. We design five types of questions to evaluate egocentric assistants’ capabilities in entity logging, event recall, task tracking, and human-centric problems (habit analysis and relationship understanding). Each example includes a multiple-choice Q&A with supporting evidence from timestamps at least 5 minutes prior to the question. Black vertical lines indicate question timestamps, while colored curved lines connect to relevant evidence timestamps.

Key Insights and Ideas

  • Personalization is a Game Changer: Fine-tuning EgoGPT on just the first day of a participant's data (Day-1) significantly boosted performance on identity-dependent tasks like RelationMap, proving that AI assistants need localized, user-specific context to be effective.
  • Audio is Essential for Egocentric AI: The pie charts in the paper's statistics reveal that a large portion of daily queries require audio understanding (e.g., identifying who said what or recognizing off-screen actions). Visual-only models miss these critical cues.
  • Hierarchical RAG Beats Naive Chunking: Trying to answer long-term queries by segmenting video into isolated 30-minute chunks leads to severe hallucinations due to a lack of global context. A top-down, multi-level summary approach preserves the narrative thread across days.

Key Results

  • SOTA on Existing Benchmarks: EgoGPT achieved state-of-the-art results on classic egocentric benchmarks, scoring 73.2% on EgoSchema (outperforming GPT-4o's 72.2%) and 61.7% on EgoThink.
  • EgoLifeQA Performance: On the challenging new EgoLifeQA benchmark, the personalized EgoGPT model achieved an average score of 36.0%, performing on par with commercial giants like GPT-4o (36.2%) despite being a fraction of the size.
  • Long-Context Mastery: For ultra-long queries requiring a look-back time greater than 24 hours, the combination of EgoGPT + EgoRAG achieved a score of 35.4%, drastically outperforming standalone implementations of Gemini-1.5-Pro (18.4%) and baseline EgoGPT (25.0%).

Limitations and Future Work

Despite its strong performance, the authors note several open challenges:

  • Audio Nuance: EgoGPT's speech understanding relies heavily on ASR-trained data, meaning it struggles to interpret human emotions, tone, and laughter.
  • Visual Identity Overfitting: Because the personalization fine-tuning happens on Day-1 data, the model tends to overfit to transient visual features, such as misidentifying a person later in the week because they changed their shirt color.
  • Single-Pass Retrieval: EgoRAG currently operates on a single-pass search mechanism. If the initial search misses the correct time window, the system cannot perform iterative, multi-step reasoning to correct its course. Future work will focus on improving error resilience and multi-step retrieval logic.

Summary updated

Abstract

We introduce EgoLife, a project to develop an egocentric life assistant that accompanies and enhances personal efficiency through AI-powered wearable glasses. To lay the foundation for this assistant, we conducted a comprehensive data collection study where six participants lived together for one week, continuously recording their daily activities - including discussions, shopping, cooking, socializing, and entertainment - using AI glasses for multimodal egocentric video capture, along with synchronized third-person-view video references. This effort resulted in the EgoLife Dataset, a comprehensive 300-hour egocentric, interpersonal, multiview, and multimodal daily life dataset with intensive annotation. Leveraging this dataset, we introduce EgoLifeQA, a suite of long-context, life-oriented question-answering tasks designed to provide meaningful assistance in daily life by addressing practical questions such as recalling past relevant events, monitoring health habits, and offering personalized recommendations. To address the key technical challenges of (1) developing robust visual-audio models for egocentric data, (2) enabling identity recognition, and (3) facilitating long-context question answering over extensive temporal information, we introduce EgoButler, an integrated system comprising EgoGPT and EgoRAG. EgoGPT is an omni-modal model trained on egocentric datasets, achieving state-of-the-art performance on egocentric video understanding. EgoRAG is a retrieval-based component that supports answering ultra-long-context questions. Our experimental studies verify their working mechanisms and reveal critical factors and bottlenecks, guiding future improvements. By releasing our datasets, models, and benchmarks, we aim to stimulate further research in egocentric AI assistants.
BibTeX
@inproceedings{yang2025egolife,
  author = {Jingkang Yang and Shuai Liu and Hongming Guo and Yuhao Dong and Xiamengwei Zhang and Sicheng Zhang and Pengyun Wang and Zitang Zhou and Binzhu Xie and Ziyue Wang and Bei Ouyang and Zhengyu Lin and Marco Cominelli and Zhongang Cai and Yuanhan Zhang and Peiyuan Zhang and Fangzhou Hong and Joerg Widmer and Francesco Gringoli and Lei Yang and Bo Li and Ziwei Liu},
  title = {{EgoLife: Towards Egocentric Life Assistant}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  doi = {10.1109/cvpr52734.2025.02690},
}
Figure