C2KD: Cross-Lingual Cross-Modal Knowledge Distillation for Multilingual Text-Video Retrieval

Andrew Rouditchenko, Yung-Sung Chuang, Nina Shvetsova, Samuel Thomas, Rogerio Feris, Brian Kingsbury, Leonid Karlinsky, David Harwath, Hilde Kuehne, James Glass

TL;DR

C2KD (Cross-Lingual Cross-Modal Knowledge Distillation) bridges the performance gap between English and non-English text-video retrieval. By training a multilingual student model to match the cross-modal similarity distributions predicted by an ensemble of frozen English-language teacher models, C2KD significantly improves retrieval accuracy across multiple languages and domains.

What Problem Does This Paper Address?

Text-video retrieval—searching for video clips using natural language queries—has seen massive improvements thanks to joint text-video embedding spaces. However, these models exhibit a stark language bias: performance on English queries consistently outpaces performance on other languages.

This discrepancy stems from two primary issues:

  1. Data Quality: Multilingual video datasets are often created by machine-translating original English captions, introducing errors and unnatural phrasing.
  2. Pre-training Imbalance: The underlying multilingual text encoders (like LaBSE or XLM-R) are exposed to vastly more English data during their initial large-scale pre-training than any other language.

To address this, the authors look to leverage the superior performance of English text-video models to guide and improve models operating in other languages.

Methods

The authors introduce C2KD, a distillation framework that transfers cross-modal knowledge from English "teacher" models to a multilingual "student" model.

Figure 1: Overview of C2KD. A multilingual student model computes text-video similarity scores for a batch of video and text inputs, while teacher models process the same video and English translations. The student is trained with two objectives. LNCE (described in Section 3.1) trains the model to have high text-video scores for text and video pairs using the cross entropy loss. LC2KD (described in Section 3.3) distills the knowledge from the teacher English text-video scores using a cross entropy loss.

During training, the student model receives video clips paired with captions in various non-English languages. Simultaneously, an ensemble of multiple pre-trained, frozen teacher models receives the same videos but with the original English versions of those captions.

The student is optimized using a dual-objective loss function:

  1. Standard Contrastive Loss ($L_{NCE}$): A standard Noise-Contrastive Estimation loss that pushes the embeddings of true text-video pairs together while pulling mismatched pairs apart.
  2. Cross-Lingual Distillation Loss ($L_{C2KD}$): A novel cross-entropy loss that forces the student's multilingual text-video similarity matrix to match a pooled similarity matrix generated by the English teachers.

Unlike prior work that uses a regression loss (like Smooth L1) to match individual similarity scores independently, C2KD applies a softmax over the entire batch. This treats the similarity scores as a probability distribution, preserving the broader context of how a query relates to all candidate videos in a batch.

Datasets

The method was evaluated across four distinct datasets:

  • Multi-MSRVTT: An open-domain video dataset translated into 8 languages.
  • VATEX: A large-scale dataset containing authentic (non-translated) English and Chinese captions.
  • RUDDER: A low-resource dataset featuring instructional videos with captions in Indic languages.
  • Multi-YouCook2 (New): Introduced in this paper, this dataset extends the popular YouCook2 cooking dataset by translating its English recipe captions into 8 other languages (Czech, French, Russian, Chinese, German, Spanish, Japanese, and Vietnamese). Cooking videos require fine-grained reasoning about specific ingredients and sequential steps, making it a challenging benchmark.
Figure 2: Multi-YouCook2 sample video clips and multilingual captions.

Key Insights and Ideas

  • Distribution Matching > Score Regression: Normalizing similarity scores into a probability distribution over the batch before calculating the distillation loss yields significantly better results than regressing individual scores. It gives the model "leeway" to recognize semantic similarities between non-ground-truth pairs.
  • The Teacher Paradox: The strongest standalone English model does not necessarily make the best teacher. For instance, SimCSE (a powerful English sentence embedding model) performed poorly as a single teacher. Instead, ensembling complementary text encoders (Distill SBERT, mBERT, and XLM-R) provided the most robust distillation signal.
  • Zero-Shot Generalization: Because the student model utilizes LaBSE (which is pre-trained on over 100 languages), C2KD allows the model to successfully retrieve videos using queries from languages it never saw during text-video training (e.g., Ukrainian or Igbo).

Key Results

C2KD consistently outperformed baseline models across all datasets:

  • Multi-MSRVTT: Achieved a +16.2% relative improvement in average Recall@1 (R@1) across all non-English languages, jumping from 19.8% to 23.0%. The largest single-language gain was in Spanish (+18.6% relative).
  • Multi-YouCook2: Boosted average R@1 from 12.1% to 12.9% (+6.6% relative), with Japanese seeing the highest improvement (+11.9% relative).
  • Qualitative Improvements: Qualitative analysis showed that the baseline model often retrieved videos matching only a single keyword from a complex query. In contrast, the C2KD-trained model successfully captured multi-object compositions (e.g., correctly identifying "beef and tofu" rather than just one or the other).

Limitations and Future Work

While C2KD successfully narrows the performance gap between English and other languages, it does not eliminate it entirely. The authors point out that the model remains bottlenecked by the quality of machine translations in the training data; for example, poor translation quality for Vietnamese on Multi-MSRVTT led to noticeably lower performance gains.

Additionally, the evaluation was limited to a maximum of 9 languages per dataset. Future work will explore scaling to massively multilingual setups and incorporating text augmentation or paraphrasing strategies to make the student model more resilient to translation artifacts.

Summary updated

Abstract

Multilingual text-video retrieval methods have improved significantly in recent years, but the performance for other languages lags behind English. We propose a Cross-Lingual Cross-Modal Knowledge Distillation method to improve multilingual text-video retrieval. Inspired by the fact that English text-video retrieval outperforms other languages, we train a student model using input text in different languages to match the cross-modal predictions from teacher models using input text in English. We propose a cross entropy based objective which forces the distribution over the student's text-video similarity scores to be similar to those of the teacher models. We introduce a new multilingual video dataset, Multi-YouCook2, by translating the English captions in the YouCook2 video dataset to 8 other languages. Our method improves multilingual text-video retrieval performance on Multi-YouCook2 and several other datasets such as Multi-MSRVTT and VATEX. We also conducted an analysis on the effectiveness of different multilingual text models as teachers. The code, models, and dataset are available at https://github.com/roudimit/c2kd.
BibTeX
@inproceedings{rouditchenko2023c,
  author = {Andrew Rouditchenko and Yung-Sung Chuang and Nina Shvetsova and Samuel Thomas and Rogerio Feris and Brian Kingsbury and Leonid Karlinsky and David Harwath and Hilde Kuehne and James Glass},
  title = {{C2KD: Cross-Lingual Cross-Modal Knowledge Distillation for Multilingual Text-Video Retrieval}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  doi = {10.1109/icassp49357.2023.10094821},
}
Figure