Introduction
Video-Language Pre-training (VLP) aims to learn joint representations of video and text, enabling tasks like retrieval and captioning. Despite significant progress, VLP faces two major hurdles: data scarcity and the resolution trade-off. High-quality video-text pairs are difficult to curate at scale compared to image-text pairs. Furthermore, standard models often rely on single-scale visual features (typically at 1/16th resolution), forcing a compromise between capturing fine-grained local details and high-level global semantics.
The HiVLP Architecture
To address these challenges, researchers at Huawei Noah’s Ark Lab introduced HiVLP, a framework centered on hierarchical interaction between visual and textual modalities.
Hierarchical Visual Feature Group (HVFG)
The core innovation is the Hierarchical Visual Feature Group (HVFG). Instead of selecting a single resolution, HiVLP extracts a pyramid of features at 1/8, 1/16, and 1/32 scales using a Swin Transformer backbone and a Light Feature Pyramid Network (FPN). These multi-scale features are injected into the cross-attention blocks of the transformer, allowing the model to interact with text using both detailed local information and broad global context simultaneously.
Multi-level Vision Contrastive (MVC) Loss
To enhance self-supervised learning without the noise often introduced by masked token reconstruction, HiVLP employs a "global-to-local" MVC loss. This objective encourages the model to align the global representation of one scale with the local patch representations of the same video from a different augmented view, fostering consistency across different levels of detail.
Training Strategy and the Diversity Hypothesis
A key finding of this work is the Diversity Hypothesis: the variety of visual concepts in the training data is often more critical than the sheer volume of video-only data. HiVLP utilizes a hybrid training strategy, pre-training on a massive pool of 114 million image-text pairs (e.g., LAION100M) alongside 2.5 million video-text pairs (e.g., WebVid-2M). This allows the model to leverage rich visual concepts from static images that generalize effectively to temporal video data.
Experimental Results
HiVLP achieves state-of-the-art (SOTA) performance across several benchmarks:
- Text-to-Video Retrieval: On MSR-VTT, the large-scale HiVLP model achieved a 43.5% Recall@1 in zero-shot settings and 50.9% R@1* after fine-tuning, significantly outperforming previous methods like BLIP.
- Video Captioning: On the MSVD dataset, the model reached a CIDEr score of 151.6, demonstrating superior information efficiency by using only 16 frames compared to previous models requiring 64.
- Robustness: Consistent gains were observed across DiDeMo and LSMDC, validating the effectiveness of hierarchical features in diverse video contexts.
Conclusion
HiVLP demonstrates that hierarchical visual processing and high-diversity hybrid training are powerful tools for video-language understanding. While processing multiple scales increases computational demands, the "Light FPN" approach keeps the model efficient. Future research may further integrate these hierarchical spatial features with more complex temporal dynamics for long-form video understanding.
Reference: Shao, B., et al. "HiVLP: Hierarchical Interactive Video-Language Pre-Training." Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
Summary updated