Video ReCap: Recursive Captioning of Hour-Long Videos

Md Mohaiminul Islam, Ngan Ho, Xitong Yang, Tushar Nagarajan, Lorenzo Torresani, Gedas BertasiusCVPR 2024 · OpenReview

Abstract

Most video captioning models are designed to process short video clips of few seconds and output text describing low-level visual concepts (e.g., objects, scenes, atomic actions). However, most real-world videos last for minutes or hours and have a complex hierarchical structure spanning different temporal granularities. We propose Video ReCap, a recursive video captioning model that can process video inputs of dramatically different lengths (from 1 second to 2 hours) and output video captions at multiple hierarchy levels. The recursive video-language architecture exploits the synergy between different video hierarchies and can process hour-long videos efficiently. We utilize a curriculum learning training scheme to learn the hierarchical structure of videos, starting from clip-level captions describing atomic actions, then focusing on segment-level descriptions, and concluding with generating summaries for hour-long videos. Furthermore, we introduce Ego4D-HCap dataset by augmenting Ego4D with 8,267 manually collected long-range video summaries. Our recursive model can flexibly generate captions at different hierarchy levels while also being useful for other complex video understanding tasks, such as VideoQA on EgoSchema.

Introduction

Standard video captioning models often suffer from "short-sightedness," typically focusing on 5-15 second clips to describe atomic actions like "a person is cutting an onion." While effective for short-form content, these models struggle with real-world videos—such as movies, sports broadcasts, or daily life recordings—that span minutes or hours. Understanding such long-form content requires navigating a complex hierarchical structure where tiny actions (atomic steps) form intermediate activities (sequences), which eventually culminate in high-level goals.

The challenges are significant: processing hours of raw video is computationally expensive, long durations are filled with redundancy, and capturing the overarching "why" behind a sequence of actions requires a sophisticated understanding of temporal structure.

The Video ReCap Framework

To address these challenges, we introduce Video ReCap, a recursive model designed to process video across three distinct temporal granularities. Instead of relying solely on raw pixels for long-range understanding, Video ReCap uses text as a semantic compression mechanism.

The Video ReCap architecture processes video at three levels: atomic clips, intermediate segments, and global summaries, using captions from lower levels to inform higher-level descriptions.

The model operates across three levels:

  1. Level 1 (Clips): Captures atomic actions in short segments (seconds).
  2. Level 2 (Segments): Coalesces clip captions into descriptions of intermediate steps (minutes).
  3. Level 3 (Summaries): Aggregates segment descriptions into a global summary (hours).

The core innovation lies in its recursive architecture. The model utilizes captions generated at a previous hierarchy level as input for the next. This allows the system to "compress" visual information into semantic text, which is significantly more efficient to process over long durations than raw video features alone. To bridge the gap between modalities, a Language Model-based alignment module (utilizing DistilBERT) maps video features and text into a joint space.

Hierarchical Data and Training

A major hurdle in long-form video understanding is the scarcity of annotated data. To facilitate this research, we introduced Ego4D-HCap, a new benchmark for hierarchical video captioning. We augmented the existing Ego4D dataset with 8,267 manually collected long-range summaries for videos spanning up to two hours.

Training follows a Hierarchical Curriculum Learning scheme. The model is trained in stages, beginning with simple clip captions and progressively moving toward complex summaries. This mirrors human perception, where we understand small actions before grasping overarching goals. To further address data scarcity, we employed an LLM (FLAN-T5) as a "teacher" to generate thousands of pseudo-summaries from short-term captions, augmenting the training set.

Key Results and Insights

Video ReCap demonstrates state-of-the-art performance across all hierarchy levels, outperforming strong baselines such as LaViLa and BLIP2.

VideoQA Breakthrough

The model's ability to reason over long temporal windows was tested on the EgoSchema benchmark, a long-form Video Question Answering (VideoQA) task involving ~15-minute videos. Video ReCap achieved 50.23% accuracy, surpassing the previous state-of-the-art (InternVideo) by a substantial 18.13%.

Text as Compression

A key insight from this work is the effectiveness of text as a "latent representation" for long-term memory. By converting video clips into captions, the model discards visual noise while retaining the semantic "gist" necessary for long-range summaries. Ablation studies confirmed that combining video features with recursive text inputs provides a significant gain (+4.83% CIDEr score) over using text alone, proving that visual and textual modalities are complementary.

Conclusion

Video ReCap represents a significant step toward understanding hour-long video content. By leveraging a recursive, hierarchical approach, it efficiently bridges the gap between atomic actions and global narratives. While challenges remain in summarizing the most complex hour-long sequences without repetition or omission, the framework provides a robust foundation for future work in real-time hierarchical captioning and interactive video-based dialogue systems.

Summary updated

Abstract

Most video captioning models are designed to process short video clips of few seconds and output text describing low-level visual concepts (e.g., objects, scenes, atomic actions). However, most real-world videos last for minutes or hours and have a complex hierarchical structure spanning different temporal granularities. We propose Video ReCap, a recursive video captioning model that can process video inputs of dramatically different lengths (from 1 second to 2 hours) and output video captions at multiple hierarchy levels. The recursive video-language architecture exploits the synergy between different video hierarchies and can process hour-long videos efficiently. We utilize a curriculum learning training scheme to learn the hierarchical structure of videos, starting from clip-level captions describing atomic actions, then focusing on segment-level descriptions, and concluding with generating summaries for hour-long videos. Furthermore, we introduce Ego4D-HCap dataset by augmenting Ego4D with 8,267 manually collected long-range video summaries. Our recursive model can flexibly generate captions at different hierarchy levels while also being useful for other complex video understanding tasks, such as VideoQA on EgoSchema. Data, code, and models are available at: https://sites.google.com/view/vidrecap
BibTeX
@inproceedings{islam2024video,
  author = {Md Mohaiminul Islam and Ngan Ho and Xitong Yang and Tushar Nagarajan and Lorenzo Torresani and Gedas Bertasius},
  title = {{Video ReCap: Recursive Captioning of Hour-Long Videos}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  url = {https://openreview.net/forum?id=bKi1QRzFQN},
}
Figure