LifelongMemory: Leveraging LLMs for Answering Queries in Long-form Egocentric Videos

Ying Wang, Yanlai Yang, Mengye Ren

Abstract

In this paper we introduce LifelongMemory, a new framework for accessing long-form egocentric videographic memory through natural language question answering and retrieval. LifelongMemory generates concise video activity descriptions of the camera wearer and leverages the zero-shot capabilities of pretrained large language models to perform reasoning over long-form video context. Furthermore, LifelongMemory uses a confidence and explanation module to produce confident, high-quality, and interpretable answers. Our approach achieves state-of-the-art performance on the EgoSchema benchmark for question answering and is highly competitive on the natural language query (NLQ) challenge of Ego4D. Code is available at https://github.com/agentic-learning-ai-lab/lifelong-memory.
BibTeX
@misc{wang2023lifelongmemory,
  author = {Ying Wang and Yanlai Yang and Mengye Ren},
  title = {{LifelongMemory: Leveraging LLMs for Answering Queries in Long-form Egocentric Videos}},
  year = {2023},
  eprint = {2312.05269},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2312.05269},
}

Loading related content...

Figure