CLOP: Video-and-Language Pre-Training with Knowledge Regularizations

Guohao Li, Hu Yang, Feng He, Zhifan Feng, Yajuan Lyu, Hua Wu, Haifeng WangACM Multimedia 2022 · arXiv

TL;DR

This paper introduces CLOP, a novel video-and-language pre-training method that leverages "structural knowledge" (entities, actions, relations) to regularize the learning of cross-modal representations. By incorporating a Structural Knowledge Prediction task and a Knowledge-Guided Contrastive Learning approach, CLOP significantly improves performance on text-video retrieval and multi-choice QA tasks, outperforming prior state-of-the-art methods.

What Problem Does This Paper Address?

Learning effective representations that bridge the gap between video and language is crucial for tasks like video understanding and retrieval. While existing methods have shown promise, they often treat video and text implicitly, failing to explicitly leverage the rich structural information present in these modalities. This "structural knowledge"—such as the entities involved, their actions, and their relationships—holds significant semantic value that, if properly incorporated, could lead to more robust and discriminative cross-modal representations. The challenge lies in effectively using this structural knowledge not just as input, but as a "regularization" to shape the learning process and improve the quality of the learned representation space.

Methods

CLOP introduces two key components to address this challenge:

  • Structural Knowledge Prediction (SKP) Proxy Task: This task aims to pull together similar video representations in the embedding space. It first extracts "structural knowledge" (entities, actions, relations) from both video and text. This extracted knowledge is then used as pseudo-labels to train a video encoder. Essentially, it teaches the model to predict these structural elements, thereby creating semantic centers that similar videos should cluster around.
  • Knowledge-Guided Contrastive Learning (KCL) with Hard Negatives: This module enhances standard contrastive learning by intelligently selecting negative samples. It employs a two-stage sampling process:
  • Instance-Level Sampling: This retrieves mutually similar samples to form batches of "hard negatives"—samples that are semantically close but distinct.
  • Batch-Level Sampling: This uses the "structural knowledge" to guide the selection of anchors for each batch. This helps to address a "distribution discrepancy" issue where common topics might be over-represented and important but less frequent topics under-represented in random sampling. By prioritizing certain "topics" (desired semantics) based on structural knowledge, KCL ensures more effective learning of subtle variations.

These two components work together to regularize the cross-modal representation space, promoting better alignment and uniformity.

Datasets

  • Pre-training: WebVid2M (2.5 million video-text pairs).
  • Downstream Tasks (Evaluation):
    • MSRVTT (text-video retrieval)
    • MSVD (text-video retrieval)
    • VATEX (text-video retrieval)
    • DiDeMo (paragraph-to-video retrieval)
    • MSRVTT multi-choice QA

Key Insights and Ideas

  • Structural Knowledge as a Regularizer: The core idea is that explicit structural information (entities, actions, relations) can act as powerful regularizers, guiding the model to learn a more semantically meaningful and discriminative cross-modal representation space, rather than just relying on implicit correlations.
  • SKP for Semantic Centers: The SKP task effectively creates "semantic centers" in the representation space, pulling similar videos together. This is a novel way to inject structured knowledge into video representation learning.
  • KCL for Fine-Grained Discrimination: The knowledge-guided hard negative sampling in KCL is crucial for pushing apart semantically similar but distinct cross-modal pairs, enabling the model to learn subtle differences that are vital for retrieval tasks.
  • Uniformity Boosts Retrieval: The analysis shows that CLOP significantly improves the "uniformity" of the representation space (how spread out the data points are), which directly correlates with better retrieval performance, even if it slightly impacts alignment. This suggests that a well-spread representation space is key for distinguishing between similar items.
  • Synergy of SKP and KCL: The SKP and KCL components are shown to be complementary, with their combined use yielding the best results, highlighting the benefit of a multi-faceted approach to knowledge integration.

Key Results

  • CLOP achieves substantial improvements across four text-video retrieval tasks and one multi-choice QA task.
  • The "[ours] full" model (CLOP with both SKP and KCL) outperforms prior state-of-the-art methods by a significant margin. For instance, on MSRVTT, it achieves 57.8% AveR and 34.0% R@1, outperforming the previous best by several percentage points.
  • Ablation studies confirm the effectiveness of both SKP and KCL components, with the full model showing the best performance.
  • Qualitative examples demonstrate CLOP's ability to distinguish subtle semantic differences between videos and their corresponding text queries.

Limitations and Future Work

  • Dataset Size: The paper notes that WebVid2M, while effective, is an order of magnitude smaller than datasets like Howto100M. Future work could explore scaling CLOP to larger datasets.
  • Computational Cost: The paper mentions that ablation studies were conducted on a subset of WebVid due to computational constraints, suggesting that full-scale training might be resource-intensive.
  • Heuristic for KCL: The knowledge-guided sampling in KCL uses a heuristic to prioritize human-action-related semantics. Future work could explore more sophisticated methods for knowledge-guided sampling.
  • Generalizability of Structural Knowledge: While the paper focuses on entities, actions, and relations, future work could explore incorporating other forms of structural knowledge or external knowledge bases to further enhance representations.

Summary updated

BibTeX
@inproceedings{li2022clop,
  author = {Guohao Li and Hu Yang and Feng He and Zhifan Feng and Yajuan Lyu and Hua Wu and Haifeng Wang},
  title = {{CLOP: Video-and-Language Pre-Training with Knowledge Regularizations}},
  booktitle = {ACM Multimedia},
  year = {2022},
}
Figure