Introduction
Recent advancements in multimodal learning have seen models designed for static images adapted for video tasks. However, these "flat" architectures—which typically concatenate subtitle tokens and video frames into a single sequence for a standard Transformer—often struggle with the complex temporal dynamics of video. They frequently lose fine-grained alignment between dialogue and visual action and fail to exploit the sequential narrative inherent in video content.
To address these limitations, we present HERO (Hierarchical EncodeR for Omni-representation), a framework designed to capture both local cross-modal fusion and global temporal context through a specialized hierarchical structure.
Hierarchical Architecture
HERO departs from flat models by employing a two-stage encoding process:
- Cross-modal Transformer: This component handles "local" fusion. It processes individual subtitle sentences alongside the specific video frames associated with their timestamps. This allows the model to learn precise grounding between words and immediate visual actions.
- Temporal Transformer: This component manages "global" context. It aggregates the processed frame representations from the first stage to understand the broader narrative flow across the entire video clip.
Novel Pre-training Tasks
Beyond standard Masked Language Modeling (MLM) and Masked Frame Modeling (MFM), HERO introduces two video-specific tasks designed to enhance temporal reasoning:
- Video-Subtitle Matching (VSM): The model predicts both global alignment (whether a subtitle belongs to a video) and local alignment (the specific timestamp of a subtitle within the video).
- Frame Order Modeling (FOM): By predicting the original chronological order of a shuffled set of frames, the model is forced to learn temporal cause-and-effect and sequential logic.
Datasets and New Benchmarks
HERO was pre-trained on a massive scale using the HowTo100M dataset (1.22M instructional videos) and a large-scale TV Dataset (21,793 clips from medical dramas and sitcoms). The inclusion of TV data allows the model to capture complex social dynamics and multi-character interactions often missing from instructional content.
Furthermore, we introduce two new benchmarks: How2R (video-moment retrieval) and How2QA (video question-answering). To ensure these benchmarks test true multimodal understanding rather than "subtitle bias," human annotators were not shown transcripts while writing queries, forcing the model to rely on visual evidence.
Experimental Results
HERO achieves new state-of-the-art (SOTA) results across six major benchmarks. Notably, on the Video Retrieval (TVR) task, HERO nearly doubled the performance of the previous leading model (XML), increasing the R@1 metric from 3.25 to 6.21.
Ablation studies confirmed that the hierarchical design and the new VSM and FOM tasks were critical to these gains. Even on video-only tasks like DiDeMo and MSR-VTT, HERO outperformed existing models, demonstrating the robustness of its learned representations.
Conclusion and Future Work
HERO demonstrates that hierarchical modeling and temporal-aware pre-training are essential for deep video+language understanding. While currently an encoder-focused model that uses global frame features, future iterations could benefit from region-level visual features to capture finer details like color or specific object attributes. HERO sets a new standard for omni-representation learning, providing a powerful foundation for a wide array of video-and-language downstream tasks.
Summary updated