Large Language Models (LLMs) are incredibly smart, but they are essentially "blind" to the temporal nuances of long-form video. While we’ve seen great strides in short video clips (5-30 seconds), real-world video—like movies or surveillance footage—spans minutes or hours. This paper introduces Fine-Detailed Video Story generation (FDVS), a framework that translates complex, redundant video data into a hierarchical textual narrative that LLMs can actually digest and reason over.
TL;DR
FDVS tackles long video understanding by converting raw footage into a structured, multi-level textual "story" (from object-level details to full-video summaries). By combining a bottom-up interpretation mechanism with aggressive semantic redundancy reduction, it achieves state-of-the-art zero-shot performance on retrieval and QA tasks while reducing storage costs by nearly 90%.
What Problem Does This Paper Address?
Understanding long videos is hard for two main reasons: intricate long-context relationships and massive redundancy.
Traditional computer vision models often rely on "backbones" that extract deep features, but these require expensive task-specific fine-tuning and struggle to maintain coherence over long durations. On the other hand, trying to feed every frame of a 10-minute video into a multimodal LLM is computationally impossible and introduces "semantic noise"—most frames in a video don't actually add new information. The authors aim to create a system that can understand long videos without needing new training data for every task.
Methods
The authors propose a Bottom-up Video Interpretation Mechanism that works in several stages:
- Efficient Segmentation: Instead of uniform sampling (which might miss a 2-second crucial event in a 2-hour video), they segment the video into clips based on keyframes.
- Visual Redundancy Reduction: Within each clip, they use CLIP features to calculate cosine similarity between frames, discarding those that are visually nearly identical.
- Multi-level Perception: For the remaining frames, they use three "atomic" foundation models:
- Object-level: Grounding DINO identifies what and where objects are.
- Temporal-level: InternVideo classifies the actions occurring.
- Scene-level: BLIP2 generates descriptive captions.
- Hierarchical Summarization: An LLM (like Vicuna) takes this raw data and writes a "chapter" for each clip. These chapters are then filtered for Textual Semantic Redundancy (removing clips that describe the same event from different angles) before a final LLM pass synthesizes them into a cohesive "Video Story."
Datasets
The researchers evaluated FDVS across eight datasets covering three distinct tasks:
- Partially Relevant Video Retrieval (PRVR): ActivityNet Captions, Charades-STA.
- Zero-shot Video Retrieval: ActivityNet Captions, MSRVTT.
- Video Question Answering (Video QA): MSRVTT-QA, ActivityNet-QA, EgoSchema (long-form egocentric video), and NExT-QA (causal/temporal reasoning).
Key Insights and Ideas
- Text as a Universal Intermediate: By converting video to text, the model bypasses the need for cross-modal alignment training. Text is highly compressed, searchable, and interpretable by both humans and machines.
- The "Aha" of Hierarchical Reasoning: The paper finds that providing both the specific clip descriptions and the global video story to the LLM significantly boosts QA accuracy. The story provides context, while the chapters provide the "receipts" for specific details.
- Semantic vs. Visual Redundancy: A camera might move (changing pixels), but the meaning of the scene stays the same. By reducing redundancy at the textual level (comparing "chapter" embeddings), the model removes "fluff" that visual-only methods would miss.
- Zero-Shot Versatility: Because the system relies on pre-trained foundation models and natural language, it requires zero fine-tuning to switch from retrieving a video of someone cooking to answering a question about why a character is angry.
Key Results
- Superior Retrieval: On ActivityNet Captions (a long-form dataset), FDVS achieved a 14.0 R@1, significantly outperforming supervised methods like MS-SL (7.1) and zero-shot competitors like VideoLlaVA (9.2).
- State-of-the-Art QA: On the challenging EgoSchema benchmark, FDVS hit 42.4% accuracy, surpassing proprietary models like GPT-4 Turbo (31.0% without vision) and Bard + ShortViViT (42.0%).
- Storage Efficiency: Compared to Clip4Clip, which stores deep features, FDVS’s textual representations reduced storage requirements by 89.6% (from 840 bytes to 87.3 bytes per video in MSRVTT).
- Inference Speed: While the initial "storytelling" phase is slower due to multiple model calls, the downstream tasks (retrieval/QA) are much faster because the LLM is only processing a few paragraphs of text rather than thousands of video tokens.
Limitations and Future Work
- Extraction Bottleneck: The "Representation Extraction" phase is currently the slowest part of the pipeline (13.22 FPS) because it involves running an object detector, action recognizer, and captioner.
- Modality Gaps: The current version focuses on visual and textual data. The authors suggest that incorporating audio perception (like speech-to-text) would further enhance the richness of the generated stories.
- Dependency on LLM Quality: The accuracy of the final understanding is tethered to the reasoning capabilities of the LLM used (e.g., upgrading from Vicuna-7B to ChatGPT-3.5 showed a clear performance jump).
Summary updated