LifelongMemory: Leveraging LLMs for Answering Queries in Egocentric Videos

Ying Wang, Yanlai Yang, Mengye Ren

Abstract

The egocentric video natural language query (NLQ) task involves localizing a temporal window in an egocentric video that provides an answer to a posed query, which has wide applications in building personalized AI assistants. Prior methods for this task have focused on improvements of network architecture and leveraging pre-training for enhanced image and video features, but have struggled with capturing long-range temporal dependencies in lengthy videos, and cumbersome end-to-end training. Motivated by recent advancements in Large Language Models (LLMs) and vision language models, we introduce LifelongMemory, a novel framework that utilizes multiple pre-trained models to answer queries from extensive egocentric video content. We address the unique challenge by employing a pre-trained captioning model to create detailed narratives of the videos. These narratives are then used to prompt a frozen LLM to generate coarse-grained temporal window predictions, which are subsequently refined using a pre-trained NLQ model. Empirical results demonstrate that our method achieves competitive performance against existing supervised end-to-end learning methods, underlining the potential of integrating multiple pre-trained multimodal large language models in complex vision-language tasks.

Introduction

Answering natural language queries in egocentric video—footage recorded from a first-person perspective—is a critical step toward building personalized AI assistants. Imagine asking, "Where did I put my glasses two hours ago?" or "What was the name of the person I talked to in the kitchen?" To respond accurately, an AI must navigate two primary hurdles:

  1. Temporal Scale: Standard video models are typically optimized for short clips. Long-form videos spanning minutes or hours contain a vast number of actions, making it computationally prohibitive for traditional architectures to maintain a coherent "memory."
  2. Reasoning Complexity: Beyond simple object detection, these queries require understanding causal sequences and human interactions. End-to-end models often lack the sophisticated reasoning capabilities and the massive labeled datasets required for such nuanced tasks.

Methodology

LifelongMemory addresses these challenges by transforming the visual "haystack" into a searchable textual log. The framework operates through a four-stage pipeline:

  1. Egocentric Video Captioning: A Multimodal LLM (e.g., LaViLa) processes short video segments to generate descriptive captions, such as "C opens the fridge" or "C talks to man B."
  2. Caption Digest: To handle noise and redundancy, this module filters uninformative descriptions and uses an LLM to merge consecutive actions into a concise summary, significantly reducing the data volume.
  3. LLM Reasoning: The condensed log is fed to a powerful LLM (e.g., GPT-4o). The model is prompted to act as a memory recall assistant, providing an answer, a coarse temporal window, a textual explanation, and a confidence score.
  4. Task-Specific Refinement: For Video QA, the system employs "Vote by Confidence" across multiple runs. For NLQ, a specialized pre-trained model refines the LLM's coarse timestamps into fine-grained intervals.
The LifelongMemory pipeline: from raw egocentric video to LLM-driven reasoning.

Experimental Evaluation

The framework was evaluated on two major benchmarks: EgoSchema, featuring over 5,000 long-term reasoning questions, and Ego4D NLQ, which requires precise temporal localization.

Key Results

  • State-of-the-Art on EgoSchema: LifelongMemory achieved 72.0% accuracy using GPT-4o, nearly doubling the performance of traditional video-based models (which typically score around 30%).
  • Competitive NLQ Performance: The framework effectively "prunes" long videos, allowing specialized models to focus on relevant segments.
  • Scaling Effects: Performance improved significantly as the underlying LLM scaled from GPT-3.5 to GPT-4o, highlighting the importance of reasoning capacity.
Qualitative example of LifelongMemory answering an EgoSchema question with reasoning and confidence scores.

Discussion and Insights

A core insight of this work is that text serves as a universal compression for visual data. By converting pixels to prose, LifelongMemory leverages the advanced reasoning of LLMs without the need for expensive end-to-end training. Furthermore, the inclusion of self-explanation and confidence scores helps the model filter hallucinations and focus on textual evidence.

However, the system remains dependent on the quality of the initial captions. If a captioning model misses a small object or misidentifies an action, the reasoning engine cannot correct the error. Future work may involve more query-aware captioning or the use of multimodal models that can ingest longer video sequences directly.

Summary updated

Abstract

The egocentric video natural language query (NLQ) task involves localizing a temporal window in an egocentric video that provides an answer to a posed query, which has wide applications in building personalized AI assistants. Prior methods for this task have focused on improvements of network architecture and leveraging pre-training for enhanced image and video features, but have struggled with capturing long-range temporal dependencies in lengthy videos, and cumbersome end-to-end training. Motivated by recent advancements in Large Language Models (LLMs) and vision language models, we introduce LifelongMemory, a novel framework that utilizes multiple pre-trained models to answer queries from extensive egocentric video content. We address the unique challenge by employing a pre-trained captioning model to create detailed narratives of the videos. These narratives are then used to prompt a frozen LLM to generate coarse-grained temporal window predictions, which are subsequently refined using a pre-trained NLQ model. Empirical results demonstrate that our method achieves competitive performance against existing supervised end-to-end learning methods, underlining the potential of integrating multiple pre-trained multimodal large language models in complex vision-language tasks. We provide a comprehensive analysis of key design decisions and hyperparameters in our pipeline, offering insights and practical guidelines.
BibTeX
@misc{wang2023lifelongmemory,
  author = {Ying Wang and Yanlai Yang and Mengye Ren},
  title = {{LifelongMemory: Leveraging LLMs for Answering Queries in Egocentric Videos}},
  year = {2023},
  eprint = {2312.05269},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2312.05269},
}
Figure