HiVLP: Hierarchical Interactive Video-Language Pre-Training

Bin Shao, Jianzhuang Liu, Renjing Pei, Songcen Xu, Peng Dai, Juwei Lu, Weimian Li, Youliang Yan

Introduction

Video-Language Pre-training (VLP) aims to learn joint representations of video and text, enabling tasks like retrieval and captioning. Despite significant progress, VLP faces two major hurdles: data scarcity and the resolution trade-off. High-quality video-text pairs are difficult to curate at scale compared to image-text pairs. Furthermore, standard models often rely on single-scale visual features (typically at 1/16th resolution), forcing a compromise between capturing fine-grained local details and high-level global semantics.

The HiVLP Architecture

To address these challenges, researchers at Huawei Noah’s Ark Lab introduced HiVLP, a framework centered on hierarchical interaction between visual and textual modalities.

Hierarchical Visual Feature Group (HVFG)

The core innovation is the Hierarchical Visual Feature Group (HVFG). Instead of selecting a single resolution, HiVLP extracts a pyramid of features at 1/8, 1/16, and 1/32 scales using a Swin Transformer backbone and a Light Feature Pyramid Network (FPN). These multi-scale features are injected into the cross-attention blocks of the transformer, allowing the model to interact with text using both detailed local information and broad global context simultaneously.

Comparison of visual feature injection strategies: (a) Coarse, (b) Fine-Grained, and (c) the proposed HVFG injecting method, alongside the internal structure of the CA block.

Multi-level Vision Contrastive (MVC) Loss

To enhance self-supervised learning without the noise often introduced by masked token reconstruction, HiVLP employs a "global-to-local" MVC loss. This objective encourages the model to align the global representation of one scale with the local patch representations of the same video from a different augmented view, fostering consistency across different levels of detail.

Training Strategy and the Diversity Hypothesis

A key finding of this work is the Diversity Hypothesis: the variety of visual concepts in the training data is often more critical than the sheer volume of video-only data. HiVLP utilizes a hybrid training strategy, pre-training on a massive pool of 114 million image-text pairs (e.g., LAION100M) alongside 2.5 million video-text pairs (e.g., WebVid-2M). This allows the model to leverage rich visual concepts from static images that generalize effectively to temporal video data.

Experimental Results

HiVLP achieves state-of-the-art (SOTA) performance across several benchmarks:

  • Text-to-Video Retrieval: On MSR-VTT, the large-scale HiVLP model achieved a 43.5% Recall@1 in zero-shot settings and 50.9% R@1* after fine-tuning, significantly outperforming previous methods like BLIP.
  • Video Captioning: On the MSVD dataset, the model reached a CIDEr score of 151.6, demonstrating superior information efficiency by using only 16 frames compared to previous models requiring 64.
  • Robustness: Consistent gains were observed across DiDeMo and LSMDC, validating the effectiveness of hierarchical features in diverse video contexts.

Conclusion

HiVLP demonstrates that hierarchical visual processing and high-diversity hybrid training are powerful tools for video-language understanding. While processing multiple scales increases computational demands, the "Light FPN" approach keeps the model efficient. Future research may further integrate these hierarchical spatial features with more complex temporal dynamics for long-form video understanding.

Reference: Shao, B., et al. "HiVLP: Hierarchical Interactive Video-Language Pre-Training." Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.

Summary updated

Abstract

Video-Language Pre-training (VLP) has become one of the most popular research topics in deep learning. However, compared to image-language pre-training, VLP has lagged far behind due to the lack of large amounts of video-text pairs. In this work, we train a VLP model with a hybrid of image-text and video-text pairs, which significantly outperforms pre-training with only the video-text pairs. Besides, existing methods usually model the cross-modal interaction using cross-attention between single-scale visual tokens and textual tokens. These visual features are either of low resolutions lacking fine-grained information, or of high resolutions without high-level semantics. To address the issue, we propose Hierarchical interactive Video-Language Pre-training (HiVLP) that efficiently uses a hierarchical visual feature group for multi-modal cross-attention during pre-training. In the hierarchical framework, low-resolution features are learned with focus on more global high-level semantic information, while high-resolution features carry fine-grained details. As a result, HiVLP has the ability to effectively learn both the global and fine-grained representations to achieve better alignment between video and text inputs. Furthermore, we design a hierarchical multi-scale vision contrastive loss for self-supervised learning to boost the interaction between them. Experimental results show that HiVLP establishes new state-of-the-art results in three downstream tasks, text-video retrieval, video-text retrieval, and video captioning.
BibTeX
@inproceedings{shao2023hivlp,
  author = {Bin Shao and Jianzhuang Liu and Renjing Pei and Songcen Xu and Peng Dai and Juwei Lu and Weimian Li and Youliang Yan},
  title = {{HiVLP: Hierarchical Interactive Video-Language Pre-Training}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.01265},
}
Figure