Abstract
The egocentric video natural language query (NLQ) task involves localizing a temporal window in an egocentric video that provides an answer to a posed query, which has wide applications in building personalized AI assistants. Prior methods for this task have focused on improvements of network architecture and leveraging pre-training for enhanced image and video features, but have struggled with capturing long-range temporal dependencies in lengthy videos, and cumbersome end-to-end training. Motivated by recent advancements in Large Language Models (LLMs) and vision language models, we introduce LifelongMemory, a novel framework that utilizes multiple pre-trained models to answer queries from extensive egocentric video content. We address the unique challenge by employing a pre-trained captioning model to create detailed narratives of the videos. These narratives are then used to prompt a frozen LLM to generate coarse-grained temporal window predictions, which are subsequently refined using a pre-trained NLQ model. Empirical results demonstrate that our method achieves competitive performance against existing supervised end-to-end learning methods, underlining the potential of integrating multiple pre-trained multimodal large language models in complex vision-language tasks.
Introduction
Answering natural language queries in egocentric video—footage recorded from a first-person perspective—is a critical step toward building personalized AI assistants. Imagine asking, "Where did I put my glasses two hours ago?" or "What was the name of the person I talked to in the kitchen?" To respond accurately, an AI must navigate two primary hurdles:
- Temporal Scale: Standard video models are typically optimized for short clips. Long-form videos spanning minutes or hours contain a vast number of actions, making it computationally prohibitive for traditional architectures to maintain a coherent "memory."
- Reasoning Complexity: Beyond simple object detection, these queries require understanding causal sequences and human interactions. End-to-end models often lack the sophisticated reasoning capabilities and the massive labeled datasets required for such nuanced tasks.
Methodology
LifelongMemory addresses these challenges by transforming the visual "haystack" into a searchable textual log. The framework operates through a four-stage pipeline:
- Egocentric Video Captioning: A Multimodal LLM (e.g., LaViLa) processes short video segments to generate descriptive captions, such as "C opens the fridge" or "C talks to man B."
- Caption Digest: To handle noise and redundancy, this module filters uninformative descriptions and uses an LLM to merge consecutive actions into a concise summary, significantly reducing the data volume.
- LLM Reasoning: The condensed log is fed to a powerful LLM (e.g., GPT-4o). The model is prompted to act as a memory recall assistant, providing an answer, a coarse temporal window, a textual explanation, and a confidence score.
- Task-Specific Refinement: For Video QA, the system employs "Vote by Confidence" across multiple runs. For NLQ, a specialized pre-trained model refines the LLM's coarse timestamps into fine-grained intervals.
Experimental Evaluation
The framework was evaluated on two major benchmarks: EgoSchema, featuring over 5,000 long-term reasoning questions, and Ego4D NLQ, which requires precise temporal localization.
Key Results
- State-of-the-Art on EgoSchema: LifelongMemory achieved 72.0% accuracy using GPT-4o, nearly doubling the performance of traditional video-based models (which typically score around 30%).
- Competitive NLQ Performance: The framework effectively "prunes" long videos, allowing specialized models to focus on relevant segments.
- Scaling Effects: Performance improved significantly as the underlying LLM scaled from GPT-3.5 to GPT-4o, highlighting the importance of reasoning capacity.
Discussion and Insights
A core insight of this work is that text serves as a universal compression for visual data. By converting pixels to prose, LifelongMemory leverages the advanced reasoning of LLMs without the need for expensive end-to-end training. Furthermore, the inclusion of self-explanation and confidence scores helps the model filter hallucinations and focus on textual evidence.
However, the system remains dependent on the quality of the initial captions. If a captioning model misses a small object or misidentifies an action, the reasoning engine cannot correct the error. Future work may involve more query-aware captioning or the use of multimodal models that can ingest longer video sequences directly.
Summary updated