WAVER: Writing-style Agnostic Text-Video Retrieval via Distilling Vision-Language Models Through Open-Vocabulary Knowledge

Huy Le, Tung Kieu, Anh Nguyen, Ngan Le

TL;DR

WAVER introduces a cross-domain knowledge distillation framework that makes text-video retrieval robust to diverse writing styles and annotator biases. By leveraging the open-vocabulary capabilities of pre-trained vision-language models (like CLIP) to build a Video Content Dictionary, it successfully distills generalized text-based knowledge into a vision-based student model, achieving state-of-the-art performance across multiple benchmarks.

What Problem Does This Paper Address?

Text-Video Retrieval (TVR) is the task of finding a specific video clip based on a textual query. While current state-of-the-art models excel at cross-modal feature matching, they operate under a flawed assumption: that video descriptions are unbiased and uniform.

In the real world, video datasets are manually annotated by thousands of different people. This introduces significant variations in writing styles, vocabulary choices, and personal perspectives for the exact same video scene. Furthermore, while large-scale pre-trained Vision-Language Models (VLMs) like CLIP possess massive open-vocabulary knowledge, standard fine-tuning approaches often overwrite or under-utilize this rich pre-trained information in a brute-force manner. WAVER addresses this gap by introducing a framework specifically designed to be "writing-style agnostic."

Methods

WAVER tackles writing-style variations by using an implicit teacher-student framework powered by cross-domain Knowledge Distillation (KD).

  1. Video Content Dictionary (VCD) Generator: To extract objective, style-agnostic concepts, the authors use spaCy to extract verb phrases (activities) from the dataset's captions. These activities are converted into text prompts (e.g., "This is a video about [activity]") and encoded using a frozen CLIP text encoder. WAVER then measures the cosine similarity between these text embeddings and the global video embeddings, selecting the top-$\kappa$ most relevant activities to form a clean, style-agnostic Video Content Dictionary ($\mathcal{D}$).
  2. Cross-domain Knowledge Distillation: The VCD acts as the "teacher," containing comprehensive, style-enriched textual knowledge. WAVER uses a cross-attention mechanism to distill this text-based knowledge directly into the visual features of the video encoder (the "student").
  3. Learning Objectives: The distilled video embeddings and the user query embeddings are projected into a shared latent space via Multi-Layer Perceptrons (MLPs) and optimized using a bidirectional InfoNCE contrastive loss function to align matching text-video pairs.
Overview framework of WAVER
Fig. 2: The details of Video Content Dictionary Generator. As described in Section 3.2, we first form a list of all activities extracted from each caption. Then we propose a matching scheme to form a set of video-related vocabularies and select the top-κ most relevant vocabularies to create a VCD.

Datasets

The framework was evaluated on four standard text-video retrieval benchmarks:

  • MSR-VTT: 10,000 videos with 20 descriptions per video.
  • MSVD: 1,970 videos with multi-description annotations.
  • VATEX: A large-scale dataset containing over 40,000 videos and 825,000 captions.
  • DiDeMo: 10,000 videos containing over 40,000 localized text descriptions.

Key Insights and Ideas

  • Implicit Teachers over Explicit Models: Unlike traditional knowledge distillation that requires maintaining a heavy, explicit teacher model during training, WAVER uses the pre-trained VLM's open-vocabulary space as an implicit teacher via the VCD.
  • The Goldilocks Zone for Activity Selection ($\kappa$): Through ablation studies, the authors found that selecting the top-$\kappa=5$ activities yielded the best performance. Setting $\kappa$ too low ($1$) lacks sufficient semantic context, while setting it too high ($9$) introduces overly specific or noisy descriptions that dilute the model's discriminative features.
  • Cross-Dataset Robustness: Remarkably, the VCD does not even need to be generated from the target dataset. Using a VCD constructed from completely different datasets (e.g., using a VATEX-derived dictionary to evaluate MSR-VTT) still yielded high accuracy with negligible performance drops, proving that the open-vocabulary knowledge captured is truly generalized.

Key Results

WAVER achieved state-of-the-art (SOTA) performance across all four evaluation benchmarks:

  • MSR-VTT: Using a ViT-B/16 backbone, WAVER achieved a Recall@1 (R@1) of 50.4%, outperforming the previous best method, TS2-Net, by 1.0 point.
  • MSVD: WAVER achieved 50.2% R@1, outperforming the runner-up X-Pool by 3.0 points.
  • VATEX: The model demonstrated an outstanding 66.5% R@1, outperforming TS2-Net by 7.4 points.

Writing-Style Agnostic Evaluation

To explicitly test resilience to stylistic bias, the authors evaluated WAVER using captions selected by four distinct human annotators with varying writing styles. WAVER demonstrated significantly lower standard deviation (Std) in retrieval performance and consistently outperformed the baseline across all annotators, validating its stability against real-world human bias.

Table 8: Performance of WAVER on 100 videos from MSR- VTT testing set conducted by four different annotators.

Limitations and Future Work

A dedicated limitations section was not discussed in the paper. However, the authors note that future work will focus on further exploring the open-vocabulary properties of VLMs to handle even more complex semantic variations and expanding the newly proposed "writing-style agnostic task" to push the boundaries of robust text-video retrieval.

Summary updated

Abstract

Text-video retrieval, a prominent sub-field within the domain of multimodal information retrieval, has witnessed remarkable growth in recent years. However, existing methods assume video scenes are consistent with unbiased descriptions. These limitations fail to align with real-world scenarios since descriptions can be influenced by annotator biases, diverse writing styles, and varying textual perspectives. To overcome the aforementioned problems, we introduce $\texttt{WAVER}$, a cross-domain knowledge distillation framework via vision-language models through open-vocabulary knowledge designed to tackle the challenge of handling different writing styles in video descriptions. $\texttt{WAVER}$ capitalizes on the open-vocabulary properties that lie in pre-trained vision-language models and employs an implicit knowledge distillation approach to transfer text-based knowledge from a teacher model to a vision-based student. Empirical studies conducted across four standard benchmark datasets, encompassing various settings, provide compelling evidence that $\texttt{WAVER}$ can achieve state-of-the-art performance in text-video retrieval task while handling writing-style variations. The code is available at: https://github.com/Fsoft-AIC/WAVER
BibTeX
@inproceedings{le2023waver,
  author = {Huy Le and Tung Kieu and Anh Nguyen and Ngan Le},
  title = {{WAVER: Writing-style Agnostic Text-Video Retrieval via Distilling Vision-Language Models Through Open-Vocabulary Knowledge}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=REn5KP5EvH},
}
Figure