ORANGE: Text-video Retrieval via Watch-time-aware Heterogeneous Graph Contrastive Learning

Yucheng Lin, Tim Chang, Yaning Chang, Jianqiang Ma, Donghui Li, Ting Peng, Zang Li, Zhiyi Zhou, Feng WangEMNLP 2023

TL;DR

This paper introduces ORANGE, a novel text-video retrieval system that leverages a heterogeneous information graph (HIG) to capture rich relational information beyond just content. By incorporating watch-time awareness and a meta-path-wise contrastive learning strategy, ORANGE significantly improves retrieval performance, especially in real-world scenarios with ambiguous queries and low-quality videos.

What Problem Does This Paper Address?

The explosion of short-video content on platforms like TikTok and YouTube has made text-video retrieval a critical feature. Existing methods primarily focus on learning representations from video content and query text, often using sophisticated matching mechanisms. However, these approaches struggle with real-world challenges: queries are frequently brief and ambiguous, and videos can be of low quality, making content-based retrieval less effective. Furthermore, distinguishing between highly relevant but semantically similar videos to find the most satisfactory one is difficult. This paper aims to address these limitations by incorporating richer contextual and relational information into the retrieval process.

Methods

ORANGE introduces several key innovations:

  • Heterogeneous Information Graph (HIG) Construction: They build a graph where nodes represent domain objects like queries, videos, tags, and entities. Edges capture various relationships, such as a query viewing a video, queries being rewritten, or videos containing certain entities. This graph integrates both content and abundant relational information from user search logs.
  • Meta-Path-Guided Heterogeneous Graph Attention Encoder: A two-stage attention mechanism (node-level and semantic-level) is employed to encode information from the HIG. Crucially, this encoder is "watch-time-aware." It explicitly injects video watch-time information into the attention calculation, recognizing it as a key indicator of user satisfaction. This watch-time awareness is applied specifically to "VIEW" edges.
  • Meta-Path-Wise Contrastive Learning: To capture complex dependencies across different semantic relationships within the HIG, a contrastive learning loss is introduced. This loss encourages the model to learn discriminative embeddings by maximizing mutual information between different views of the same node derived from various meta-paths.
  • Graph Distillation for Online Deployment: For handling unseen queries (those not present in the constructed HIG), a graph distillation strategy is used. A BERT-based query encoder is distilled from the full ORANGE model, enabling efficient on-the-fly inference of query embeddings.

Datasets

  • Training Dataset: An industrial dataset derived from Tencent Video search logs, collected from July 1st to August 30th, 2022. This dataset was used to construct the HIG, comprising approximately 26 million nodes and 215 million edges. This dataset is notable for including watch-time and relational information, which are often missing in public academic datasets.
  • Evaluation Dataset: Collected from logs on August 31st, divided into two subsets:
  • Subset-1: Queries present in the constructed HIG.
  • Subset-2: Unseen, typically long-tail, queries.

Key Insights and Ideas

  • Beyond Content: The paper strongly advocates for the importance of relational information (user behavior, query rewriting, etc.) in addition to content for effective text-video retrieval, especially in real-world industrial settings.
  • Watch-Time as a Proxy for Satisfaction: Explicitly incorporating video watch-time into the attention mechanism is shown to be a powerful way to improve retrieval quality, as watch-time is a direct indicator of user engagement and satisfaction.
  • Heterogeneous Graphs for Richness: The HIG framework effectively unifies diverse types of information (queries, videos, entities, tags) and their complex interrelationships, providing a more comprehensive representation than traditional methods.
  • Meta-Paths Unlock Semantics: Different meta-paths within the HIG reveal distinct semantic relationships, and leveraging these through attention and contrastive learning enhances the learned embeddings.

Key Results

  • Offline Evaluation (Subset-1): ORANGE significantly outperforms baseline models like SBERT, CLIP4Clip, and GAT. It achieves an 8.9%, 7.3%, and 5.6% improvement on R@K metrics (K=10, 50, 100) and an 18.3% improvement on AWT@100 compared to GAT.
  • Offline Evaluation (Subset-2 - Unseen Queries): The distilled ORANGE model (ORANGEdst) shows competitive performance, slightly outperforming SBERT and CLIP4Clip, and is comparable to GAT.
  • Online A/B Test: The system, when deployed in a real-world online video-search service, demonstrated a statistically significant cumulative improvement of 2.08% in terms of Average Watch Time (AWT).

Limitations and Future Work

  • Pre-defined Meta-Paths: The reliance on pre-defined meta-paths and their fixed length (set to 3) might limit the model's expressive capability. The authors suggest designing an automatic method to identify useful meta-paths.
  • Performance Drop for Unseen Queries: While the distilled version handles unseen queries, it still exhibits a performance decline compared to the full model. Future work could explore incrementally constructing the HIG more frequently (e.g., hourly instead of daily) to better incorporate newly acquired user data.
  • Efficient Graph Distillation: The authors plan to explore more efficient graph distillation strategies for future work.
  • Informative Graph Encoders: Investigating graph encoders that can better consider abundant search behavior information is also a future direction.

Summary updated

BibTeX
@inproceedings{lin2023orange,
  author = {Yucheng Lin and Tim Chang and Yaning Chang and Jianqiang Ma and Donghui Li and Ting Peng and Zang Li and Zhiyi Zhou and Feng Wang},
  title = {{ORANGE: Text-video Retrieval via Watch-time-aware Heterogeneous Graph Contrastive Learning}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
}
Figure