Leveraging Textual Memory and Key Frame Reasoning for Full Video Understanding Using Off-the-Shelf LLMs and VLMs (Student Abstract)

Harsh Dubey, Chulwoo PackAAAI 2025 · DOI

TL;DR

This paper introduces a novel approach to video understanding by combining off-the-shelf Large Language Models (LLMs) and Vision-Language Models (VLMs) with a "textual memory" mechanism. This method aims to improve fine-grained comprehension and temporal coherence in videos, addressing limitations of current models in handling complex, real-world scenarios. They also propose a new evaluation metric, VAD-Score, to better assess video description quality.

What Problem Does This Paper Address?

Current Large-scale Video-Language Models (LVLMs) often struggle with understanding complex videos, especially those with multiple subjects, events, or non-standard perspectives. They tend to perform well on benchmarks in a zero-shot manner but fail to generalize to real-world videos. A key challenge is the lack of fine-grained understanding and long-term temporal memory, leading to incomplete video analyses. This paper aims to bridge this gap by enhancing generalization and temporal coherence in video understanding.

Methods

The proposed approach leverages a combination of a Vision-Language Model (VLM) and a Large Language Model (LLM) with a novel "textual memory" mechanism.

  • Key Frame Processing and Attribute Reasoning: They use LLAVA 1.6, a VLM, to process key frames from a video. For each key frame, LLAVA generates a detailed "Frame Description" (FD) and a set of six "Query/Answer" (Q/A) pairs. These Q/A pairs are designed to encourage the model to "think out loud" about the frame's content, extracting deeper attribute-specific information about objects, background, and scene elements.
  • Textual Memory and Contextual Continuity: To address the lack of long-term temporal memory, they introduce a "textual memory" mechanism. The comprehensive frame descriptions (CFDs) generated from the FDs and Q/A pairs are fed as textual memory into the LLM. This creates a memory chain, ensuring that the description of each subsequent frame builds upon the information from the previous one, maintaining coherence across the entire video sequence.
  • Iterative Video Description Generation: The model iteratively combines the generated CFDs to produce a final video description. This process starts by combining the first two CFDs and then progressively incorporates each subsequent CFD, ensuring that critical details from all key frames are included in the final narrative.
  • VAD-Score (Video Automated Description Score): A new evaluation metric is introduced to assess the quality and granularity of video descriptions. It calculates precision, recall, and F1 scores for events, subjects, and objects, then combines them using a weighted average (0.5 for events, 0.3 for subjects, 0.2 for objects), prioritizing event comprehension.

Datasets

  • DREAM-1K: The model was evaluated on 10 randomly sampled videos from the DREAM-1K dataset. This dataset spans categories like Movie Animation, Movie Live Action, Shorts, Stock, and YouTube, including both short and long videos.
  • Human-Annotated Ground Truth: Due to limited annotations in DREAM-1K, the authors manually created detailed descriptions for the sampled videos, capturing events, subjects, and objects to provide a more comprehensive ground truth for evaluation.

Key Insights and Ideas

  • Leveraging Off-the-Shelf VLMs for Generalization: By using a VLM instead of a full LVLM, the authors aim to improve generalization to unseen videos, as VLMs have shown better performance in this regard.
  • "Thinking Out Loud" for Deeper Understanding: The use of Q/A pairs to prompt the VLM encourages a more analytical reasoning process for each frame, going beyond simple visual description to capture nuanced attributes and contextual understanding.
  • Textual Memory as a Coherence Bridge: The textual memory mechanism is a crucial innovation for maintaining temporal continuity. By feeding previous frame summaries into the LLM, the model can build a coherent narrative across frames, overcoming the limitations of short-term context windows.
  • VAD-Score for Granular Evaluation: The VAD-Score provides a more nuanced evaluation than standard metrics by explicitly weighting the importance of events, subjects, and objects, reflecting a more holistic understanding of video content.

Key Results

  • Overall Performance: The model achieved competitive results on the DREAM-1K dataset.
  • Video Length: The model performed better on short videos (VAD-Score of 0.692) compared to long videos (0.595), suggesting that shorter sequences with fewer events and simpler transitions are easier to maintain coherence for.
  • Subject Recognition: Subject recognition was consistently strong across both video lengths, often achieving perfect F1 scores.
  • Live Action Performance: In live-action videos, the model achieved an event F1 score of 0.75, comparable to state-of-the-art models like GPT-4V and Gemini 1.5 Pro.
  • Areas for Improvement: The model lagged behind in Movie Animation and YouTube categories, with F1 scores of 0.29. However, it showed respectable F1 scores for Shorts (0.65) and Stock (0.60) videos.

Limitations and Future Work

  • Performance Discrepancies: The paper notes that the model performed less well on animation and YouTube content compared to live-action, shorts, and stock videos. This suggests potential biases or limitations in how the VLM and LLM handle different visual styles and content types.
  • Future Work: The authors suggest that future work will involve in-depth analysis at scale, implying a need for more extensive experimentation and potentially model refinement to address the observed performance gaps.

Summary updated

BibTeX
@inproceedings{dubey2025leveraging,
  author = {Harsh Dubey and Chulwoo Pack},
  title = {{Leveraging Textual Memory and Key Frame Reasoning for Full Video Understanding Using Off-the-Shelf LLMs and VLMs (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i28.35248},
}

Loading related content...

Figure