TL;DR
This paper introduces QUALITY, a novel query-aware pre-training model that effectively fuses visual, textual, and query information for video search relevance ranking. By addressing the limitations of existing models that primarily focus on text or struggle with the query-video mismatch, QUALITY significantly improves video search performance.
What Problem Does This Paper Address?
In the realm of video search, accurately ranking videos based on user queries is paramount. Traditional methods often rely heavily on text, neglecting the rich visual and multi-modal information present in videos. While recent advancements in Vision-Language Pre-training (VLP) have shown promise, they often fall short for video search due to pre-training objectives that don't align with the nuances of query-video relevance. Specifically, VLP models trained on general image-text pairs may not effectively capture the relationship between concise search queries and complex video content. Furthermore, these models often overlook valuable video-specific information like keyframes and tags. This paper tackles the challenge of bridging this gap by developing a model that can comprehensively understand and rank video relevance by explicitly considering the query's intent alongside multi-modal video content.
Methods
The proposed solution, QUALITY (QUery-Aware pre-training model with Multi-modaLITY), operates in a two-step framework:
- Query-aware Pre-training Model (QUALITY): This model extends the dual-tower structure of existing VLP models. It incorporates three key towers: a query tower, a video visual tower, and a video text tower.
- Multi-modal Input: It processes uniformly sampled video frames, video titles, tags, and the input query.
- AdaptFormer: To enable efficient training and prevent catastrophic forgetting of pre-trained uni-modal encoders (BERT and ViT), a lightweight "AdaptFormer" module is employed, freezing most pre-trained parameters.
- Tag Guidance: A tag-guided cross-attention network aligns semantic tag information with visual features, enriching the visual representation.
- Query-awareness: This is a core innovation. Instead of just aligning video content with descriptions, QUALITY explicitly models the relationship between the query and video frames through a Vision-Query Contrastive (VQC) task and a Vision-Query Matching (VQM) task. This addresses the mismatch between concise queries and verbose video descriptions.
- Pre-training Objectives: QUALITY is pre-trained using five objectives: Vision-Query Contrastive Learning (VQC), Vision-Text Contrastive Learning (VTC), Vision-Query Matching (VQM), Vision-Text Matching (VTM), and Masked Language Modeling (MLM). VQM and VTM are enhanced with both online and offline hard negative mining strategies.
- Multi-modal Based Ranking Relevance Model: This model integrates the pre-trained QUALITY model with a ranking loss function.
- Feature Extraction: It utilizes embeddings from QUALITY (query, text, visual) and handcrafted numerical features (e.g., BM25, click similarity).
- Cross-Encoder: A pre-trained transformer-based cross-encoder processes query and video text (title, actors, tags) to generate a semantic relevance representation.
- Ordinal Regression Loss: To balance calibration (stable predictions) and ranking ability, the paper proposes an ordinal regression-based loss function. This approach models relevance as a K-grade problem, accommodating labeled order and thresholds. A binary cross-entropy loss is also incorporated to improve differentiation between relevant and irrelevant candidates.
Datasets
- Training QUALITY: A dataset of 10 million videos from Tencent Video, including keyframes, titles, and over 15,000 labeled tags.
- Training and Testing Ranking Relevance: Manually annotated query-video pairs from video search logs. This resulted in a training set of 270,000 pairs and a test set of over 90,000 items. Relevance grades range from 1 (Bad) to 4 (Excellent).
Key Insights and Ideas
- Query-Centric Pre-training is Crucial: Explicitly modeling the relationship between the query and video content (not just video descriptions) is key to improving relevance ranking in real-world search scenarios. This directly addresses the "query-video mismatch."
- Tag Information Enhances Visual Understanding: Integrating video tags as guidance helps the model develop a semantic understanding of video content beyond low-level visual features.
- Ordinal Regression Balances Calibration and Ranking: The proposed ordinal regression loss function effectively balances the ability to make stable, interpretable predictions (calibration) with the ability to order items correctly (ranking), a common trade-off in ranking tasks.
- AdaptFormer for Efficient Pre-training: Using AdaptFormer significantly speeds up training and preserves knowledge from large pre-trained models, making it practical for real-world applications.
- Hard Mining is Effective: Both online and offline hard mining strategies for queries and negatives are crucial for robust pre-training, especially for VQM and VTM tasks.
Key Results
- Pre-training (QUALITY): Achieved an AUC of 0.683 and a PNR of 2.180, outperforming a baseline Video-ALBEF model by 10.5% AUC and 42.2% PNR.
- Ranking Relevance Model:
- The proposed ordinal regression loss function achieved an AUC of 0.921 and a PNR of 8.914, outperforming pointwise loss (0.925 AUC, 6.911 PNR) and Combined-Pair loss (0.918 AUC, 8.565 PNR). The ordinal loss shows a better balance between calibration and ranking.
- Integrating QUALITY into the ranking model improved performance over text-based baselines, achieving an AUC of 0.921 and a PNR of 8.914.
- Ablation Studies: Demonstrated the significant contribution of each component: query tower (+0.1% AUC, +0.8% PNR), tag guidance (+0.1% AUC, +0.5% PNR), and AdaptFormer (+0.2% AUC, +1.5% PNR) when compared to a text-based baseline.
- Online A/B Testing: The deployed model achieved a 2.1% improvement in Average Watch Time (AWT) and a 5.7% improvement in AGSB (satisfaction gain) compared to online baselines.
Limitations and Future Work
- Noise in Graph Construction: The paper acknowledges that the accuracy of their graph mining strategy for identifying unobserved query-video pairs might be influenced by noise in the graph construction.
- Future Work: The authors suggest investigating alternative hard mining strategies or visual debiasing strategies to enhance performance and mitigate potential noise issues.
Summary updated