Towards Long Video Understanding via Fine-detailed Video Story Generation

Zeng You, Zhiquan Wen, Yaofo Chen, Xin Li, Runhao Zeng, Yaowei Wang, Mingkui Tan

Abstract

Long video understanding has become a critical task in computer vision, driving advancements across numerous applications from surveillance to content retrieval. Existing video understanding methods suffer from two challenges when dealing with long video understanding: intricate long-context relationship modeling and interference from redundancy. To tackle these challenges, we introduce Fine-Detailed Video Story generation (FDVS), which interprets long videos into detailed textual representations. Specifically, to achieve fine-grained modeling of long-temporal content, we propose a Bottom-up Video Interpretation Mechanism that progressively interprets video content from clips to video. To avoid interference from redundant information in videos, we introduce a Semantic Redundancy Reduction mechanism that removes redundancy at both the visual and textual levels. Our method transforms long videos into hierarchical textual representations that contain multi-granularity information of the video. With these representations, FDVS is applicable to various tasks without any fine-tuning. We evaluate the proposed method across eight datasets spanning three tasks. The performance demonstrates the effectiveness and versatility of our method.

Introduction

Large Language Models (LLMs) have demonstrated remarkable reasoning capabilities, yet they remain largely "blind" to the temporal nuances of long-form video. While significant progress has been made in short video clip analysis (5-30 seconds), real-world applications—such as movie analysis or surveillance—require understanding footage that spans minutes or hours. Understanding long videos is inherently difficult due to two primary factors: intricate long-context relationships and massive data redundancy.

Traditional computer vision models often rely on deep feature backbones that require expensive task-specific fine-tuning and struggle with long-term coherence. Conversely, feeding every frame of a long video into a multimodal LLM is computationally prohibitive and introduces significant "semantic noise." To address these gaps, we propose Fine-Detailed Video Story generation (FDVS), a framework that translates complex, redundant video data into a structured, hierarchical textual narrative that LLMs can effectively process and reason over.

Conceptual overview of FDVS, showing how object, temporal, and scene-level information are combined to generate a fine-detailed video story.

Methodology: The FDVS Framework

The core of FDVS is a Bottom-up Video Interpretation Mechanism that progressively builds a narrative from individual clips to a complete video story. The pipeline consists of four key stages:

  1. Efficient Segmentation: Instead of uniform sampling, which may overlook brief but crucial events, the video is segmented into clips based on keyframes.
  2. Visual Redundancy Reduction: Within each clip, CLIP features are used to calculate cosine similarity between frames. Visually near-identical frames are discarded to streamline processing.
  3. Multi-level Perception: For the remaining frames, three specialized foundation models extract "atomic" information: object-level (Grounding DINO), temporal-level (InternVideo), and scene-level (BLIP2).
  4. Hierarchical Summarization: An LLM (e.g., Vicuna) processes this raw data to write a "chapter" for each clip. These chapters then undergo Textual Semantic Redundancy Reduction before a final LLM pass synthesizes them into a cohesive "Long Video Story."
The FDVS pipeline overview, illustrating the transition from raw video clips to a long video story through visual/textual redundancy reduction and hierarchical LLM processing.

Experiments and Results

We evaluated FDVS across eight datasets covering three distinct tasks: Partially Relevant Video Retrieval (PRVR), Zero-shot Video Retrieval, and Video Question Answering (Video QA).

Key Performance Metrics

  • Superior Retrieval: On the ActivityNet Captions dataset, FDVS achieved a 14.0 R@1, significantly outperforming supervised methods like MS-SL (7.1) and zero-shot competitors like VideoLlaVA (9.2).
  • State-of-the-Art QA: On the challenging EgoSchema benchmark for long-form egocentric video, FDVS reached 42.4% accuracy, surpassing proprietary models like GPT-4 Turbo (31.0% without vision) and Bard + ShortViViT (42.0%).
  • Storage Efficiency: By using textual representations instead of deep features, FDVS reduced storage requirements by 89.6% compared to methods like Clip4Clip (from 840 bytes to 87.3 bytes per video in MSRVTT).

Discussion and Conclusion

The success of FDVS highlights the power of Text as a Universal Intermediate. Converting video to text bypasses the need for complex cross-modal alignment training while providing a highly compressed, searchable, and interpretable format. Furthermore, the hierarchical reasoning approach—providing both specific clip descriptions and a global video story—significantly boosts QA accuracy by offering both context and detail.

In conclusion, FDVS offers a robust, zero-shot framework for long video understanding. By transforming raw pixels into structured stories, it enables LLMs to reason over long-temporal contexts with high efficiency. Future work will explore incorporating audio perception to further enhance the richness of the generated stories.

Summary updated

Abstract

Long video understanding has become a critical task in computer vision, driving advancements across numerous applications from surveillance to content retrieval. Existing video understanding methods suffer from two challenges when dealing with long video understanding: intricate long-context relationship modeling and interference from redundancy. To tackle these challenges, we introduce Fine-Detailed Video Story generation (FDVS), which interprets long videos into detailed textual representations. Specifically, to achieve fine-grained modeling of long-temporal content, we propose a Bottom-up Video Interpretation Mechanism that progressively interprets video content from clips to video. To avoid interference from redundant information in videos, we introduce a Semantic Redundancy Reduction mechanism that removes redundancy at both the visual and textual levels. Our method transforms long videos into hierarchical textual representations that contain multi-granularity information of the video. With these representations, FDVS is applicable to various tasks without any fine-tuning. We evaluate the proposed method across eight datasets spanning three tasks. The performance demonstrates the effectiveness and versatility of our method.
BibTeX
@article{you2025towards,
  author = {Zeng You and Zhiquan Wen and Yaofo Chen and Xin Li and Runhao Zeng and Yaowei Wang and Mingkui Tan},
  title = {{Towards Long Video Understanding via Fine-detailed Video Story Generation}},
  journal = {IEEE Trans. Circuits Syst. Video Technol.},
  year = {2025},
  url = {https://openreview.net/forum?id=jUsQ9pW6Nj},
}
Figure