HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training

Linjie Li, Yen-Chun Chen, Yu Cheng, Zhe Gan, Licheng Yu, Jingjing Liu

Introduction

Large-scale pre-training has fundamentally changed Natural Language Processing (NLP) and image-text modeling. However, extending these successes to the video domain presents unique challenges. Existing video-language models often suffer from three primary limitations:

  1. Flat Architectures: Concatenating subtitles and video frames into a single sequence fails to capture the hierarchical temporal alignment between specific sentences and their corresponding visual frames.
  2. Generic Tasks: Borrowing pre-training tasks directly from image-text research ignores the sequential flow and global context inherent in video.
  3. Limited Data Domains: A heavy reliance on instructional videos (e.g., HowTo100M) limits a model's ability to handle complex social dynamics and multi-character interactions.

HERO (Hierarchical EncodeR for Omni-representation) addresses these issues through a novel nested transformer architecture and temporal-aware pre-training tasks.

Architecture and Methodology

The core of HERO is its Hierarchical Encoder, designed to mirror the multi-level structure of video content.

Overview of the HERO architecture and its four pre-training tasks.

The architecture consists of two main components:

  • Cross-modal Transformer (Local): This layer fuses a subtitle sentence with its corresponding local video frames, capturing immediate visual-textual context.
  • Temporal Transformer (Global): The outputs from the local step are processed by a global transformer that considers the entire video clip, enabling the model to understand the broader narrative and long-range temporal dependencies.

Pre-training Tasks

HERO is trained using four distinct tasks to learn robust representations:

  • Masked Language Modeling (MLM) & Masked Frame Modeling (MFM): Standard reconstruction tasks where the model predicts missing words or visual frames.
  • Video-Subtitle Matching (VSM): A novel task requiring the model to predict both global alignment (matching a subtitle to a video) and local alignment (identifying the specific timestamp of a subtitle).
  • Frame Order Modeling (FOM): The model must reconstruct the original chronological order of a shuffled set of frames, forcing it to learn sequential logic.

Datasets and Benchmarks

HERO was pre-trained on a massive corpus of 7.6 million video clips, combining the instructional HowTo100M dataset with a TV Dataset featuring medical dramas and sitcoms. The latter provides the social complexity necessary for real-world applications. Additionally, the authors introduced two new benchmarks, How2R (video-moment retrieval) and How2QA (video question-answering), featuring human-written queries to ensure high quality and avoid biases found in automatically generated captions.

Experimental Results

HERO achieves state-of-the-art (SOTA) performance across six major benchmarks, demonstrating significant improvements over previous models.

Comparison of HERO against SOTA baselines across multiple video-language benchmarks.

Key findings include:

  • Superior Retrieval: On the TVR benchmark, HERO nearly doubled the performance of the previous best model (XML) in R@1 metrics (6.21 vs 3.25).
  • Effective QA: In TVQA, HERO outperformed the SOTA baseline by over 3%, reaching 73.61% accuracy without requiring specialized region-level features.
  • Hierarchy and Temporal Reasoning: Ablation studies confirmed that the hierarchical design significantly outperforms "flat" encoders. Furthermore, the FOM task proved essential for tasks requiring temporal reasoning, such as Video QA.

Conclusion

HERO demonstrates that explicitly modeling the hierarchical and sequential nature of video is crucial for effective video-language pre-training. While the current model uses global frame features, future work may incorporate region-level spatial grounding and more powerful generative decoders for video captioning. HERO sets a new standard for omni-representation learning, bridging the gap between moving images and natural language.

Summary updated

Abstract

We present HERO, a novel framework for large-scale video+language omni-representation learning. HERO encodes multimodal inputs in a hierarchical structure, where local context of a video frame is captured by a Cross-modal Transformer via multimodal fusion, and global video context is captured by a Temporal Transformer. In addition to standard Masked Language Modeling (MLM) and Masked Frame Modeling (MFM) objectives, we design two new pre-training tasks: (i) Video-Subtitle Matching (VSM), where the model predicts both global and local temporal alignment; and (ii) Frame Order Modeling (FOM), where the model predicts the right order of shuffled video frames. HERO is jointly trained on HowTo100M and large-scale TV datasets to gain deep understanding of complex social dynamics with multi-character interactions. Comprehensive experiments demonstrate that HERO achieves new state of the art on multiple benchmarks over Text-based Video/Video-moment Retrieval, Video Question Answering (QA), Video-and-language Inference and Video Captioning tasks across different domains. We also introduce two new challenging benchmarks How2QA and How2R for Video QA and Retrieval, collected from diverse video content over multimodalities.
BibTeX
@inproceedings{li2020hero,
  author = {Linjie Li and Yen-Chun Chen and Yu Cheng and Zhe Gan and Licheng Yu and Jingjing Liu},
  title = {{HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=PG1yl3UuNRP},
}
Figure