TL;DR
WAVER introduces a cross-domain knowledge distillation framework that makes text-video retrieval robust to diverse writing styles and annotator biases. By leveraging the open-vocabulary capabilities of pre-trained vision-language models (like CLIP) to build a Video Content Dictionary, it successfully distills generalized text-based knowledge into a vision-based student model, achieving state-of-the-art performance across multiple benchmarks.
What Problem Does This Paper Address?
Text-Video Retrieval (TVR) is the task of finding a specific video clip based on a textual query. While current state-of-the-art models excel at cross-modal feature matching, they operate under a flawed assumption: that video descriptions are unbiased and uniform.
In the real world, video datasets are manually annotated by thousands of different people. This introduces significant variations in writing styles, vocabulary choices, and personal perspectives for the exact same video scene. Furthermore, while large-scale pre-trained Vision-Language Models (VLMs) like CLIP possess massive open-vocabulary knowledge, standard fine-tuning approaches often overwrite or under-utilize this rich pre-trained information in a brute-force manner. WAVER addresses this gap by introducing a framework specifically designed to be "writing-style agnostic."
Methods
WAVER tackles writing-style variations by using an implicit teacher-student framework powered by cross-domain Knowledge Distillation (KD).
- Video Content Dictionary (VCD) Generator: To extract objective, style-agnostic concepts, the authors use
spaCyto extract verb phrases (activities) from the dataset's captions. These activities are converted into text prompts (e.g., "This is a video about [activity]") and encoded using a frozen CLIP text encoder. WAVER then measures the cosine similarity between these text embeddings and the global video embeddings, selecting the top-$\kappa$ most relevant activities to form a clean, style-agnostic Video Content Dictionary ($\mathcal{D}$). - Cross-domain Knowledge Distillation: The VCD acts as the "teacher," containing comprehensive, style-enriched textual knowledge. WAVER uses a cross-attention mechanism to distill this text-based knowledge directly into the visual features of the video encoder (the "student").
- Learning Objectives: The distilled video embeddings and the user query embeddings are projected into a shared latent space via Multi-Layer Perceptrons (MLPs) and optimized using a bidirectional InfoNCE contrastive loss function to align matching text-video pairs.
Datasets
The framework was evaluated on four standard text-video retrieval benchmarks:
- MSR-VTT: 10,000 videos with 20 descriptions per video.
- MSVD: 1,970 videos with multi-description annotations.
- VATEX: A large-scale dataset containing over 40,000 videos and 825,000 captions.
- DiDeMo: 10,000 videos containing over 40,000 localized text descriptions.
Key Insights and Ideas
- Implicit Teachers over Explicit Models: Unlike traditional knowledge distillation that requires maintaining a heavy, explicit teacher model during training, WAVER uses the pre-trained VLM's open-vocabulary space as an implicit teacher via the VCD.
- The Goldilocks Zone for Activity Selection ($\kappa$): Through ablation studies, the authors found that selecting the top-$\kappa=5$ activities yielded the best performance. Setting $\kappa$ too low ($1$) lacks sufficient semantic context, while setting it too high ($9$) introduces overly specific or noisy descriptions that dilute the model's discriminative features.
- Cross-Dataset Robustness: Remarkably, the VCD does not even need to be generated from the target dataset. Using a VCD constructed from completely different datasets (e.g., using a VATEX-derived dictionary to evaluate MSR-VTT) still yielded high accuracy with negligible performance drops, proving that the open-vocabulary knowledge captured is truly generalized.
Key Results
WAVER achieved state-of-the-art (SOTA) performance across all four evaluation benchmarks:
- MSR-VTT: Using a ViT-B/16 backbone, WAVER achieved a Recall@1 (R@1) of 50.4%, outperforming the previous best method, TS2-Net, by 1.0 point.
- MSVD: WAVER achieved 50.2% R@1, outperforming the runner-up X-Pool by 3.0 points.
- VATEX: The model demonstrated an outstanding 66.5% R@1, outperforming TS2-Net by 7.4 points.
Writing-Style Agnostic Evaluation
To explicitly test resilience to stylistic bias, the authors evaluated WAVER using captions selected by four distinct human annotators with varying writing styles. WAVER demonstrated significantly lower standard deviation (Std) in retrieval performance and consistently outperformed the baseline across all annotators, validating its stability against real-world human bias.
Limitations and Future Work
A dedicated limitations section was not discussed in the paper. However, the authors note that future work will focus on further exploring the open-vocabulary properties of VLMs to handle even more complex semantic variations and expanding the newly proposed "writing-style agnostic task" to push the boundaries of robust text-video retrieval.
Summary updated