HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training

Linjie Li, Yen-Chun Chen, Yu Cheng, Zhe Gan, Licheng Yu, Jingjing Liu

Introduction

Recent advancements in multimodal learning have seen models designed for static images adapted for video tasks. However, these "flat" architectures—which typically concatenate subtitle tokens and video frames into a single sequence for a standard Transformer—often struggle with the complex temporal dynamics of video. They frequently lose fine-grained alignment between dialogue and visual action and fail to exploit the sequential narrative inherent in video content.

To address these limitations, we present HERO (Hierarchical EncodeR for Omni-representation), a framework designed to capture both local cross-modal fusion and global temporal context through a specialized hierarchical structure.

Hierarchical Architecture

HERO departs from flat models by employing a two-stage encoding process:

  1. Cross-modal Transformer: This component handles "local" fusion. It processes individual subtitle sentences alongside the specific video frames associated with their timestamps. This allows the model to learn precise grounding between words and immediate visual actions.
  2. Temporal Transformer: This component manages "global" context. It aggregates the processed frame representations from the first stage to understand the broader narrative flow across the entire video clip.
Overview of the HERO framework, illustrating the hierarchical structure and the four pre-training tasks: MFM, VSM, MLM, and FOM.

Novel Pre-training Tasks

Beyond standard Masked Language Modeling (MLM) and Masked Frame Modeling (MFM), HERO introduces two video-specific tasks designed to enhance temporal reasoning:

  • Video-Subtitle Matching (VSM): The model predicts both global alignment (whether a subtitle belongs to a video) and local alignment (the specific timestamp of a subtitle within the video).
  • Frame Order Modeling (FOM): By predicting the original chronological order of a shuffled set of frames, the model is forced to learn temporal cause-and-effect and sequential logic.

Datasets and New Benchmarks

HERO was pre-trained on a massive scale using the HowTo100M dataset (1.22M instructional videos) and a large-scale TV Dataset (21,793 clips from medical dramas and sitcoms). The inclusion of TV data allows the model to capture complex social dynamics and multi-character interactions often missing from instructional content.

Furthermore, we introduce two new benchmarks: How2R (video-moment retrieval) and How2QA (video question-answering). To ensure these benchmarks test true multimodal understanding rather than "subtitle bias," human annotators were not shown transcripts while writing queries, forcing the model to rely on visual evidence.

Experimental Results

HERO achieves new state-of-the-art (SOTA) results across six major benchmarks. Notably, on the Video Retrieval (TVR) task, HERO nearly doubled the performance of the previous leading model (XML), increasing the R@1 metric from 3.25 to 6.21.

Comparison of HERO against SOTA baselines across multiple benchmarks including TVR, How2R, TVQA, VIOLIN, and TVC.

Ablation studies confirmed that the hierarchical design and the new VSM and FOM tasks were critical to these gains. Even on video-only tasks like DiDeMo and MSR-VTT, HERO outperformed existing models, demonstrating the robustness of its learned representations.

Conclusion and Future Work

HERO demonstrates that hierarchical modeling and temporal-aware pre-training are essential for deep video+language understanding. While currently an encoder-focused model that uses global frame features, future iterations could benefit from region-level visual features to capture finer details like color or specific object attributes. HERO sets a new standard for omni-representation learning, providing a powerful foundation for a wide array of video-and-language downstream tasks.

Summary updated

Abstract

We present HERO, a novel framework for large-scale video+language omni-representation learning. HERO encodes multimodal inputs in a hierarchical structure, where local context of a video frame is captured by a Cross-modal Transformer via multimodal fusion, and global video context is captured by a Temporal Transformer. In addition to standard Masked Language Modeling (MLM) and Masked Frame Modeling (MFM) objectives, we design two new pre-training tasks: (i) Video-Subtitle Matching (VSM), where the model predicts both global and local temporal alignment; and (ii) Frame Order Modeling (FOM), where the model predicts the right order of shuffled video frames. HERO is jointly trained on HowTo100M and large-scale TV datasets to gain deep understanding of complex social dynamics with multi-character interactions. Comprehensive experiments demonstrate that HERO achieves new state of the art on multiple benchmarks over Text-based Video/Video-moment Retrieval, Video Question Answering (QA), Video-and-language Inference and Video Captioning tasks across different domains. We also introduce two new challenging benchmarks How2QA and How2R for Video QA and Retrieval, collected from diverse video content over multimodalities.
BibTeX
@inproceedings{li2020hero,
  author = {Linjie Li and Yen-Chun Chen and Yu Cheng and Zhe Gan and Licheng Yu and Jingjing Liu},
  title = {{HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=PG1yl3UuNRP},
}
Figure