TL;DR
The EgoLife project introduces a massive 300-hour, multi-person, multimodal egocentric dataset captured over a full week to pave the way for true AI life assistants. Alongside this dataset, the authors present the EgoLifeQA benchmark for long-context daily reasoning and EgoButler, an AI agent that combines an omni-modal video-audio model (EgoGPT) with a hierarchical retrieval system (EgoRAG) to successfully answer questions spanning days of continuous footage.
What Problem Does This Paper Address?
While egocentric (first-person) vision has advanced rapidly with datasets like Epic-Kitchens and Ego4D, existing resources suffer from two major limitations: they feature short recording durations and focus almost exclusively on a single person's isolated perspective. Real life, however, is continuous and deeply social. Current models cannot track long-term personal habits or understand the complex multi-person dynamics of a shared living space.
To bridge this gap, the authors created a fully instrumented environment where multiple participants lived together for a week, capturing the rich, messy reality of daily life—including multi-person conversations, shared cooking, chores, and socializing.
Methods
The authors developed EgoButler, a dual-system agent designed to handle clip-level understanding and long-horizon retrieval:
- EgoGPT (System-I): A lightweight (7B) omni-modal vision-audio-language model built on top of LLaVA-OneVision. To handle the critical audio component of egocentric data, the authors integrated a dedicated audio branch using Whisper Large v3. EgoGPT continuously processes 30-second video clips to generate dense visual-audio captions. Crucially, the model can be personalized by fine-tuning on a participant's first day of data to improve identity-aware tracking.
- EgoRAG (System-II): Standard video models cannot ingest a week's worth of video frames at once. EgoRAG solves this by building a hierarchical structured memory bank. Captions generated by EgoGPT are summarized by a language model at hourly and daily levels. When a user asks a question, EgoRAG performs a top-down search (Day $\rightarrow$ Hour $\rightarrow$ Clip) using a scoring function that balances visual and textual relevance to retrieve the top-3 most relevant 30-second clips for EgoGPT to analyze.
Datasets
The paper introduces and utilizes three distinct datasets:
- EgoLife Dataset: A 300-hour dataset tracking 6 participants living together for 7 days. Each participant wore Meta Aria glasses (recording egocentric video, audio, IMU, and gaze). The environment was augmented with 15 synchronized third-person GoPro cameras and 2 mmWave radar devices for spatial tracking.
- EgoLifeQA Benchmark: A curated set of 3,000 high-quality, multiple-choice question-answering pairs (500 per participant). The questions require a "look-back" time of at least 5 minutes (and up to multiple days) and are split into five real-world tasks: EntityLog (object tracking/pricing), EventRecall (past events), HabitInsight (personal habits), RelationMap (social interactions), and TaskMaster (task management).
- EgoIT-99K: A newly curated dataset of 99,480 QA pairs derived from 9 classic egocentric datasets, used to instruction-tune EgoGPT on first-person perspectives before deployment.
Key Insights and Ideas
- Personalization is a Game Changer: Fine-tuning EgoGPT on just the first day of a participant's data (Day-1) significantly boosted performance on identity-dependent tasks like RelationMap, proving that AI assistants need localized, user-specific context to be effective.
- Audio is Essential for Egocentric AI: The pie charts in the paper's statistics reveal that a large portion of daily queries require audio understanding (e.g., identifying who said what or recognizing off-screen actions). Visual-only models miss these critical cues.
- Hierarchical RAG Beats Naive Chunking: Trying to answer long-term queries by segmenting video into isolated 30-minute chunks leads to severe hallucinations due to a lack of global context. A top-down, multi-level summary approach preserves the narrative thread across days.
Key Results
- SOTA on Existing Benchmarks: EgoGPT achieved state-of-the-art results on classic egocentric benchmarks, scoring 73.2% on EgoSchema (outperforming GPT-4o's 72.2%) and 61.7% on EgoThink.
- EgoLifeQA Performance: On the challenging new EgoLifeQA benchmark, the personalized EgoGPT model achieved an average score of 36.0%, performing on par with commercial giants like GPT-4o (36.2%) despite being a fraction of the size.
- Long-Context Mastery: For ultra-long queries requiring a look-back time greater than 24 hours, the combination of EgoGPT + EgoRAG achieved a score of 35.4%, drastically outperforming standalone implementations of Gemini-1.5-Pro (18.4%) and baseline EgoGPT (25.0%).
Limitations and Future Work
Despite its strong performance, the authors note several open challenges:
- Audio Nuance: EgoGPT's speech understanding relies heavily on ASR-trained data, meaning it struggles to interpret human emotions, tone, and laughter.
- Visual Identity Overfitting: Because the personalization fine-tuning happens on Day-1 data, the model tends to overfit to transient visual features, such as misidentifying a person later in the week because they changed their shirt color.
- Single-Pass Retrieval: EgoRAG currently operates on a single-pass search mechanism. If the initial search misses the correct time window, the system cannot perform iterative, multi-step reasoning to correct its course. Future work will focus on improving error resilience and multi-step retrieval logic.
Summary updated