CBVS: A Large-Scale Chinese Image-Text Benchmark for Real-World Short Video Search Scenarios

Xiangshuo Qiao, Xianxin Li, Xiaozhe Qu, Jie Zhang, Yang Liu, Yu Luo, Cihang Jin, Jin Ma

Abstract

In the rapidly evolving landscape of short video platforms, effective search relies on aligning user queries with video covers. However, these covers often differ significantly from natural images, featuring complex "mashups" of visual elements and embedded text. This paper introduces CBVS, a large-scale Chinese image-text benchmark specifically tailored for short video search. We also propose UniCLIP, a model that leverages auxiliary "agent tasks" during training to learn from cover text without requiring expensive OCR at inference time. UniCLIP achieves state-of-the-art performance on the CBVS-20K benchmark and has been successfully deployed in production environments.

Traditional vision-language models like CLIP are typically trained on open-domain datasets consisting of natural photographs. In contrast, short video covers (e.g., on platforms like TikTok or WeChat Channels) are often artificial compositions. These "mashups" include spliced graphics, vibrant backgrounds, and crucial cover text added by creators to summarize content.

Comparison between open-domain images (left) and short video cover images (right), highlighting the distinct visual morphology and presence of embedded text.

Existing datasets fail to capture this unique morphology. Furthermore, while embedded text provides vital semantic clues, relying on real-time OCR (Optical Character Recognition) engines during search is computationally prohibitive and fails when text is missing. The authors set out to bridge this gap by providing a massive, domain-specific dataset and a model architecture that learns from cover text without becoming dependent on it.

The CBVS Benchmark

To address the domain gap, the researchers present the Chinese Benchmark for Video Search (CBVS) suite, which includes:

  • CBVS-5M/10M: Large-scale unsupervised datasets containing 5 million and 10 million <video title, video cover> pairs, respectively. These include OCR-extracted text from the covers to facilitate training.
  • CBVS-20K: A high-quality, manually labeled benchmark featuring 20,000 real user query-cover pairs. Human experts graded the relevance on a three-point scale to ensure rigorous evaluation.
  • Broad Diversity: The data spans 32 categories, including Film, Education, Game, and Commodity, ensuring the model generalizes across various topics.

UniCLIP: OCR-Aware Training, OCR-Free Inference

The proposed UniCLIP architecture is designed to be "OCR-aware" during training but "OCR-free" at inference. It builds on the standard CLIP framework but adds two novel auxiliary "agent tasks":

  1. Presence-guided Encoder: Trains the visual encoder to recognize whether an image contains cover text.
  2. Semantic-guided Encoder: Encourages the visual encoder to extract semantic meaning from embedded text by matching visual tokens against actual OCR-extracted strings.

Crucially, these auxiliary encoders are only used during training. When deployed for real-world search, they are discarded. This results in a visual encoder that has "learned" to pay attention to text-heavy areas and understand their context while functioning as a standard, high-speed image encoder during live queries.

Key Results and Impact

UniCLIP achieved state-of-the-art (SOTA) results on the CBVS-20K benchmark, outperforming major competitors like Wukong, Taiyi-CLIP, and Ernie-ViL2.0. Key metrics include:

  • Performance Boost: Compared to the baseline QA-CLIP, fine-tuning on the CBVS dataset improved the Recall@1 (R@1) by 18.25% and the Positive-to-Negative Ratio (PNR) by 3.67%.
  • Real-World Deployment: The model is already operational within Tencent’s online video search systems, handling hundreds of millions of visits and delivering significant practical gains in search relevance.

Conclusion

CBVS and UniCLIP bridge the gap between general-purpose vision models and the specific requirements of short video search. By treating text as a visual feature to be learned rather than a separate modality to be parsed at runtime, UniCLIP offers a scalable and efficient solution for real-world multi-modal retrieval.

Summary updated

BibTeX
@inproceedings{qiao2024cbvs,
  author = {Xiangshuo Qiao and Xianxin Li and Xiaozhe Qu and Jie Zhang and Yang Liu and Yu Luo and Cihang Jin and Jin Ma},
  title = {{CBVS: A Large-Scale Chinese Image-Text Benchmark for Real-World Short Video Search Scenarios}},
  booktitle = {MMSR @ CIKM},
  year = {2024},
}
Figure