Toward Long Video Understanding via Fine-Detailed Video Story Generation

Zeng You, Zhiquan Wen, Yaofo Chen, Xin Li, Runhao Zeng, Yaowei Wang, Mingkui Tan

Large Language Models (LLMs) are incredibly smart, but they are essentially "blind" to the temporal nuances of long-form video. While we’ve seen great strides in short video clips (5-30 seconds), real-world video—like movies or surveillance footage—spans minutes or hours. This paper introduces Fine-Detailed Video Story generation (FDVS), a framework that translates complex, redundant video data into a hierarchical textual narrative that LLMs can actually digest and reason over.

TL;DR

FDVS tackles long video understanding by converting raw footage into a structured, multi-level textual "story" (from object-level details to full-video summaries). By combining a bottom-up interpretation mechanism with aggressive semantic redundancy reduction, it achieves state-of-the-art zero-shot performance on retrieval and QA tasks while reducing storage costs by nearly 90%.

What Problem Does This Paper Address?

Understanding long videos is hard for two main reasons: intricate long-context relationships and massive redundancy.

Traditional computer vision models often rely on "backbones" that extract deep features, but these require expensive task-specific fine-tuning and struggle to maintain coherence over long durations. On the other hand, trying to feed every frame of a 10-minute video into a multimodal LLM is computationally impossible and introduces "semantic noise"—most frames in a video don't actually add new information. The authors aim to create a system that can understand long videos without needing new training data for every task.

Methods

The authors propose a Bottom-up Video Interpretation Mechanism that works in several stages:

  1. Efficient Segmentation: Instead of uniform sampling (which might miss a 2-second crucial event in a 2-hour video), they segment the video into clips based on keyframes.
  2. Visual Redundancy Reduction: Within each clip, they use CLIP features to calculate cosine similarity between frames, discarding those that are visually nearly identical.
  3. Multi-level Perception: For the remaining frames, they use three "atomic" foundation models:
  4. Object-level: Grounding DINO identifies what and where objects are.
  5. Temporal-level: InternVideo classifies the actions occurring.
  6. Scene-level: BLIP2 generates descriptive captions.
  7. Hierarchical Summarization: An LLM (like Vicuna) takes this raw data and writes a "chapter" for each clip. These chapters are then filtered for Textual Semantic Redundancy (removing clips that describe the same event from different angles) before a final LLM pass synthesizes them into a cohesive "Video Story."

Datasets

The researchers evaluated FDVS across eight datasets covering three distinct tasks:

  • Partially Relevant Video Retrieval (PRVR): ActivityNet Captions, Charades-STA.
  • Zero-shot Video Retrieval: ActivityNet Captions, MSRVTT.
  • Video Question Answering (Video QA): MSRVTT-QA, ActivityNet-QA, EgoSchema (long-form egocentric video), and NExT-QA (causal/temporal reasoning).

Key Insights and Ideas

  • Text as a Universal Intermediate: By converting video to text, the model bypasses the need for cross-modal alignment training. Text is highly compressed, searchable, and interpretable by both humans and machines.
  • The "Aha" of Hierarchical Reasoning: The paper finds that providing both the specific clip descriptions and the global video story to the LLM significantly boosts QA accuracy. The story provides context, while the chapters provide the "receipts" for specific details.
  • Semantic vs. Visual Redundancy: A camera might move (changing pixels), but the meaning of the scene stays the same. By reducing redundancy at the textual level (comparing "chapter" embeddings), the model removes "fluff" that visual-only methods would miss.
  • Zero-Shot Versatility: Because the system relies on pre-trained foundation models and natural language, it requires zero fine-tuning to switch from retrieving a video of someone cooking to answering a question about why a character is angry.

Key Results

  • Superior Retrieval: On ActivityNet Captions (a long-form dataset), FDVS achieved a 14.0 R@1, significantly outperforming supervised methods like MS-SL (7.1) and zero-shot competitors like VideoLlaVA (9.2).
  • State-of-the-Art QA: On the challenging EgoSchema benchmark, FDVS hit 42.4% accuracy, surpassing proprietary models like GPT-4 Turbo (31.0% without vision) and Bard + ShortViViT (42.0%).
  • Storage Efficiency: Compared to Clip4Clip, which stores deep features, FDVS’s textual representations reduced storage requirements by 89.6% (from 840 bytes to 87.3 bytes per video in MSRVTT).
  • Inference Speed: While the initial "storytelling" phase is slower due to multiple model calls, the downstream tasks (retrieval/QA) are much faster because the LLM is only processing a few paragraphs of text rather than thousands of video tokens.

Limitations and Future Work

  • Extraction Bottleneck: The "Representation Extraction" phase is currently the slowest part of the pipeline (13.22 FPS) because it involves running an object detector, action recognizer, and captioner.
  • Modality Gaps: The current version focuses on visual and textual data. The authors suggest that incorporating audio perception (like speech-to-text) would further enhance the richness of the generated stories.
  • Dependency on LLM Quality: The accuracy of the final understanding is tethered to the reasoning capabilities of the LLM used (e.g., upgrading from Vicuna-7B to ChatGPT-3.5 showed a clear performance jump).

Summary updated

Abstract

Long video understanding has become a critical task in computer vision, driving advancements across numerous applications from surveillance to content retrieval. Existing video understanding methods suffer from two challenges when dealing with long video understanding: intricate long-context relationship modeling and interference from redundancy. To tackle these challenges, we introduce Fine-Detailed Video Story generation (FDVS), which interprets long videos into detailed textual representations. Specifically, to achieve fine-grained modeling of long-temporal content, we propose a Bottom-up Video Interpretation Mechanism that progressively interprets video content from clips to video. To avoid interference from redundant information in videos, we introduce a Semantic Redundancy Reduction mechanism that removes redundancy at both the visual and textual levels. Our method transforms long videos into hierarchical textual representations that contain multi-granularity information of the video. With these representations, FDVS is applicable to various tasks without any fine-tuning. We evaluate the proposed method across eight datasets spanning three tasks. The performance demonstrates the effectiveness and versatility of our method.
BibTeX
@article{you2025toward,
  author = {Zeng You and Zhiquan Wen and Yaofo Chen and Xin Li and Runhao Zeng and Yaowei Wang and Mingkui Tan},
  title = {{Toward Long Video Understanding via Fine-Detailed Video Story Generation}},
  journal = {IEEE Trans. Circuits Syst. Video Technol.},
  year = {2025},
  url = {https://openreview.net/forum?id=jUsQ9pW6Nj},
}

Loading related content...

Figure