TL;DR
C2KD (Cross-Lingual Cross-Modal Knowledge Distillation) bridges the performance gap between English and non-English text-video retrieval. By training a multilingual student model to match the cross-modal similarity distributions predicted by an ensemble of frozen English-language teacher models, C2KD significantly improves retrieval accuracy across multiple languages and domains.
What Problem Does This Paper Address?
Text-video retrieval—searching for video clips using natural language queries—has seen massive improvements thanks to joint text-video embedding spaces. However, these models exhibit a stark language bias: performance on English queries consistently outpaces performance on other languages.
This discrepancy stems from two primary issues:
- Data Quality: Multilingual video datasets are often created by machine-translating original English captions, introducing errors and unnatural phrasing.
- Pre-training Imbalance: The underlying multilingual text encoders (like LaBSE or XLM-R) are exposed to vastly more English data during their initial large-scale pre-training than any other language.
To address this, the authors look to leverage the superior performance of English text-video models to guide and improve models operating in other languages.
Methods
The authors introduce C2KD, a distillation framework that transfers cross-modal knowledge from English "teacher" models to a multilingual "student" model.
During training, the student model receives video clips paired with captions in various non-English languages. Simultaneously, an ensemble of multiple pre-trained, frozen teacher models receives the same videos but with the original English versions of those captions.
The student is optimized using a dual-objective loss function:
- Standard Contrastive Loss ($L_{NCE}$): A standard Noise-Contrastive Estimation loss that pushes the embeddings of true text-video pairs together while pulling mismatched pairs apart.
- Cross-Lingual Distillation Loss ($L_{C2KD}$): A novel cross-entropy loss that forces the student's multilingual text-video similarity matrix to match a pooled similarity matrix generated by the English teachers.
Unlike prior work that uses a regression loss (like Smooth L1) to match individual similarity scores independently, C2KD applies a softmax over the entire batch. This treats the similarity scores as a probability distribution, preserving the broader context of how a query relates to all candidate videos in a batch.
Datasets
The method was evaluated across four distinct datasets:
- Multi-MSRVTT: An open-domain video dataset translated into 8 languages.
- VATEX: A large-scale dataset containing authentic (non-translated) English and Chinese captions.
- RUDDER: A low-resource dataset featuring instructional videos with captions in Indic languages.
- Multi-YouCook2 (New): Introduced in this paper, this dataset extends the popular YouCook2 cooking dataset by translating its English recipe captions into 8 other languages (Czech, French, Russian, Chinese, German, Spanish, Japanese, and Vietnamese). Cooking videos require fine-grained reasoning about specific ingredients and sequential steps, making it a challenging benchmark.
Key Insights and Ideas
- Distribution Matching > Score Regression: Normalizing similarity scores into a probability distribution over the batch before calculating the distillation loss yields significantly better results than regressing individual scores. It gives the model "leeway" to recognize semantic similarities between non-ground-truth pairs.
- The Teacher Paradox: The strongest standalone English model does not necessarily make the best teacher. For instance, SimCSE (a powerful English sentence embedding model) performed poorly as a single teacher. Instead, ensembling complementary text encoders (Distill SBERT, mBERT, and XLM-R) provided the most robust distillation signal.
- Zero-Shot Generalization: Because the student model utilizes LaBSE (which is pre-trained on over 100 languages), C2KD allows the model to successfully retrieve videos using queries from languages it never saw during text-video training (e.g., Ukrainian or Igbo).
Key Results
C2KD consistently outperformed baseline models across all datasets:
- Multi-MSRVTT: Achieved a +16.2% relative improvement in average Recall@1 (R@1) across all non-English languages, jumping from 19.8% to 23.0%. The largest single-language gain was in Spanish (+18.6% relative).
- Multi-YouCook2: Boosted average R@1 from 12.1% to 12.9% (+6.6% relative), with Japanese seeing the highest improvement (+11.9% relative).
- Qualitative Improvements: Qualitative analysis showed that the baseline model often retrieved videos matching only a single keyword from a complex query. In contrast, the C2KD-trained model successfully captured multi-object compositions (e.g., correctly identifying "beef and tofu" rather than just one or the other).
Limitations and Future Work
While C2KD successfully narrows the performance gap between English and other languages, it does not eliminate it entirely. The authors point out that the model remains bottlenecked by the quality of machine translations in the training data; for example, poor translation quality for Vietnamese on Multi-MSRVTT led to noticeably lower performance gains.
Additionally, the evaluation was limited to a maximum of 9 languages per dataset. Future work will explore scaling to massively multilingual setups and incorporating text augmentation or paraphrasing strategies to make the student model more resilient to translation artifacts.
Summary updated