Introduction
Large-scale pre-training has fundamentally changed Natural Language Processing (NLP) and image-text modeling. However, extending these successes to the video domain presents unique challenges. Existing video-language models often suffer from three primary limitations:
- Flat Architectures: Concatenating subtitles and video frames into a single sequence fails to capture the hierarchical temporal alignment between specific sentences and their corresponding visual frames.
- Generic Tasks: Borrowing pre-training tasks directly from image-text research ignores the sequential flow and global context inherent in video.
- Limited Data Domains: A heavy reliance on instructional videos (e.g., HowTo100M) limits a model's ability to handle complex social dynamics and multi-character interactions.
HERO (Hierarchical EncodeR for Omni-representation) addresses these issues through a novel nested transformer architecture and temporal-aware pre-training tasks.
Architecture and Methodology
The core of HERO is its Hierarchical Encoder, designed to mirror the multi-level structure of video content.
The architecture consists of two main components:
- Cross-modal Transformer (Local): This layer fuses a subtitle sentence with its corresponding local video frames, capturing immediate visual-textual context.
- Temporal Transformer (Global): The outputs from the local step are processed by a global transformer that considers the entire video clip, enabling the model to understand the broader narrative and long-range temporal dependencies.
Pre-training Tasks
HERO is trained using four distinct tasks to learn robust representations:
- Masked Language Modeling (MLM) & Masked Frame Modeling (MFM): Standard reconstruction tasks where the model predicts missing words or visual frames.
- Video-Subtitle Matching (VSM): A novel task requiring the model to predict both global alignment (matching a subtitle to a video) and local alignment (identifying the specific timestamp of a subtitle).
- Frame Order Modeling (FOM): The model must reconstruct the original chronological order of a shuffled set of frames, forcing it to learn sequential logic.
Datasets and Benchmarks
HERO was pre-trained on a massive corpus of 7.6 million video clips, combining the instructional HowTo100M dataset with a TV Dataset featuring medical dramas and sitcoms. The latter provides the social complexity necessary for real-world applications. Additionally, the authors introduced two new benchmarks, How2R (video-moment retrieval) and How2QA (video question-answering), featuring human-written queries to ensure high quality and avoid biases found in automatically generated captions.
Experimental Results
HERO achieves state-of-the-art (SOTA) performance across six major benchmarks, demonstrating significant improvements over previous models.
Key findings include:
- Superior Retrieval: On the TVR benchmark, HERO nearly doubled the performance of the previous best model (XML) in R@1 metrics (6.21 vs 3.25).
- Effective QA: In TVQA, HERO outperformed the SOTA baseline by over 3%, reaching 73.61% accuracy without requiring specialized region-level features.
- Hierarchy and Temporal Reasoning: Ablation studies confirmed that the hierarchical design significantly outperforms "flat" encoders. Furthermore, the FOM task proved essential for tasks requiring temporal reasoning, such as Video QA.
Conclusion
HERO demonstrates that explicitly modeling the hierarchical and sequential nature of video is crucial for effective video-language pre-training. While the current model uses global frame features, future work may incorporate region-level spatial grounding and more powerful generative decoders for video captioning. HERO sets a new standard for omni-representation learning, bridging the gap between moving images and natural language.
Summary updated