CBVS: A Large-Scale Chinese Image-Text Benchmark for Real-World Short Video Search Scenarios

Xiangshuo Qiao, Xianxin Li, Xiaozhe Qu, Jie Zhang, Yang Liu, Yu Luo, Cihang Jin, Jin Ma

TL;DR

This paper introduces CBVS, the first large-scale Chinese image-text benchmark specifically for short video search, addressing the limitations of existing datasets that focus on general images. They also propose UniCLIP, a novel OCR-free method that effectively integrates cover text semantics into image-text contrastive learning, achieving state-of-the-art results on their new benchmark.

What Problem Does This Paper Address?

Current vision-language models, like CLIP, have shown great promise in tasks like image retrieval. However, these models are typically trained on large datasets of general images (e.g., animals, buildings) with corresponding text descriptions. This is a significant departure from real-world scenarios like short video search, where the primary visual representation is a "cover image." These cover images are user-generated, often visually distinct from typical open-domain images, and frequently accompanied by text overlays (OCR text) that provide crucial semantic context. Existing Chinese image-text datasets are largely composed of general images, failing to capture the unique characteristics of short video covers and their associated text. This gap hinders the development of effective models for short video search.

Methods

The core of the paper's methodological contribution lies in UniCLIP, a novel approach designed to leverage cover text information without relying on explicit OCR during inference. UniCLIP builds upon the CLIP framework but introduces two "agent tasks" during training:

  • Presence-guided Encoder: This task trains the model to predict whether a cover image contains text. This encourages the visual encoder to pay attention to potential text regions.
  • Semantic-guided Encoder: This task trains the model to determine if a given text matches the semantic content of the cover image (specifically, the OCR text if present). This helps the model learn to incorporate the meaning of the cover text.

Crucially, UniCLIP is designed to be OCR-free during inference. This means that while the training process uses OCR text to guide the model, the final deployed model does not require an OCR module, making it robust to missing text and reducing computational overhead. The training loss is a combination of standard Image-Text Contrastive (ITC) loss and the losses from these two agent tasks.

Datasets

The paper introduces the CBVS (Chinese Video-Text Benchmark for Short Video Search) dataset, which is presented in three versions:

  • CBVS-20K: A manually fine-labeled dataset containing 20,000 high-quality <user query-video cover> pairs. This serves as a benchmark for evaluating image-text matching in short video search. It includes user queries from browser logs and human annotations for relevance.
  • CBVS-5M and CBVS-10M: Large-scale, unsupervised datasets containing approximately 5 million and 10 million <video title-video cover> pairs, respectively. These are intended for pre-training and fine-tuning visual-language models.

CBVS is highlighted as the largest publicly available Chinese image-text dataset specifically providing cover images.

Key Insights and Ideas

  • Domain Shift is Critical: The paper strongly emphasizes that short video cover images are morphologically different from open-domain images. Models trained on general image-text data do not transfer well to this specific domain, underscoring the need for specialized datasets like CBVS.
  • Cover Text is a Rich Source of Semantics: The presence of text overlays on video covers is not just decorative; it's a vital semantic complement to the visual information. Effectively integrating this text is key to improving search performance.
  • OCR-Free Inference is Achievable and Beneficial: UniCLIP demonstrates that it's possible to leverage cover text during training without being dependent on OCR at inference time. This makes the model more robust and efficient.
  • Agent Tasks for Guided Learning: The introduction of "presence-guided" and "semantic-guided" encoders as auxiliary tasks during training is an innovative way to steer the model's attention and learning towards relevant aspects of the cover images and their associated text.

Key Results

  • UniCLIP achieves state-of-the-art performance on the CBVS-20K benchmark across various metrics, including Recall@1 (50.3%), MR (0.692), PNR (3.069), and NDCG@10 (0.893).
  • Fine-tuning existing models like QA-CLIP and CN-CLIP on the CBVS-5M dataset leads to significant improvements, demonstrating the value of the dataset itself. For instance, fine-tuning QA-CLIP on CBVS-5M resulted in an 18.25% improvement in R@1.
  • UniCLIP outperforms other advanced Chinese image-text models on CBVS-20K, often by a considerable margin, particularly in metrics like MR (e.g., UniCLIP's 0.692 vs. Taiyi-CLIP's 0.407).
  • The ablation study shows that both the presence-guided and semantic-guided encoders contribute positively to UniCLIP's performance, with the semantic-guided encoder playing a more crucial role in capturing semantic information.

Limitations and Future Work

  • Domain Generalization: While UniCLIP performs well on CBVS, the authors acknowledge that balancing the specific video search domain with broader generalization remains an area for exploration.
  • Data Distribution Bias: Although efforts were made to mitigate data distribution bias, the inherent nature of user-generated content might still present challenges.
  • Future Directions: The authors suggest extending CBVS to other downstream tasks like title generation. They also propose further research into multi-modal fusion within the CLIP framework, inspired by UniCLIP's approach. They also encourage incorporating larger visual backbones (like ViT-L) and larger training corpora (like CBVS-10M) to further enhance performance.

Summary updated

BibTeX
@inproceedings{qiao2024cbvs,
  author = {Xiangshuo Qiao and Xianxin Li and Xiaozhe Qu and Jie Zhang and Yang Liu and Yu Luo and Cihang Jin and Jin Ma},
  title = {{CBVS: A Large-Scale Chinese Image-Text Benchmark for Real-World Short Video Search Scenarios}},
  booktitle = {MMSR @ CIKM},
  year = {2024},
}
Figure