MDMMT: Multidomain Multimodal Transformer for Video Retrieval

Maksim Dzabraev, Maksim Kalashnikov, Stepan Komkov, Aleksandr Petiushko

Abstract

We present a new state-of-the-art on the text-to-video retrieval task on MSRVTT and LSMDC benchmarks where our model outperforms all previous solutions by a large margin. Moreover, state-of-the-art results are achieved using a single model and without finetuning. This multidomain generalisation is achieved by a proper combination of different video caption datasets. We show that our practical approach for training on different datasets can improve test results of each other. Additionally, we check intersection between many popular datasets and show that MSRVTT as well as ActivityNet contains a significant overlap between the test and the training parts.

Introduction

The goal of text-to-video retrieval is to identify the most relevant video segment in a large library based on a text query. Despite its importance, the field faces three major hurdles: the trade-off between scalability and accuracy, narrow domain specialization, and compromised benchmark integrity due to data leakage. MDMMT (Multidomain Multimodal Transformer) addresses these by training a single, robust model across multiple diverse datasets simultaneously, setting a new standard for performance and generalization.

Methodology: Two-Stream Architecture

A core design choice in video retrieval is the architecture type. "Single-stream" models process text and video together, offering high accuracy but suffering from slow inference speeds as the entire network must run for every query-video pair. In contrast, "two-stream" models generate separate embeddings for text and video, allowing for efficient large-scale retrieval via simple similarity metrics.

Comparison of single-stream (left) vs. two-stream (right) architectures. MDMMT utilizes the efficient two-stream approach.

MDMMT builds upon the Multi Modal Transformer (MMT) architecture using a "Late Fusion" strategy. Instead of raw pixels, it leverages pretrained "expert" networks to extract features for Motion, RGB, and Audio. The authors found that using advanced motion experts like CLIP and irCSN152 significantly outperformed previous standards. These features are then processed by a Transformer encoder to capture temporal dependencies.

Multidomain Data Strategy

To ensure generalization across diverse content—from movie clips to YouTube vlogs—MDMMT is trained on a "mega-dataset" combined from various sources. This multidomain approach prevents the model from becoming a "one-trick pony" specialized for a single benchmark.

Visualization of the diverse datasets used, showing the relationship between total video duration and the number of unique captions.

The training mix includes benchmarks like MSRVTT and ActivityNet alongside datasets like TwitterVines, YouCook2, and HowTo100M. Crucially, the authors introduced a "clean" version of MSRVTT ($M_c$) by manually removing training videos that overlapped with the test set, addressing the widespread issue of data leakage.

Key Insights and Results

The research yields several critical insights:

  • Generalization over Specialization: A single model trained on diverse data can outperform models fine-tuned on specific benchmarks. This suggests that broad exposure is more effective than narrow optimization.
  • Motion is the Foundation: The choice of motion expert is the most critical factor for retrieval performance; the transformer backbone cannot compensate for poor initial feature extraction.
  • Data Leakage: The study revealed that ~37% of the MSRVTT 1k-A test set and ~10% of the ActivityNet validation set overlap with training data, inflating previous results and calling for more rigorous data splits.

MDMMT achieved state-of-the-art results across major benchmarks:

  • MSRVTT (1k-A): 38.9% Recall@1 (previous best: 31.2%).
  • LSMDC: 18.8% R@1 (previous best: 12.9%).
  • ActivityNet: 20.1% R@1, more than doubling previous baselines like CLIP (9.4%).

Conclusion

MDMMT demonstrates that multidomain training and high-quality motion experts are key to robust video retrieval. While the model shows exceptional zero-shot capabilities, future work could involve automating the dataset sampling process and further investigating data leakage in massive datasets like HowTo100M.

Summary updated

Abstract

We present a new state-of-the-art on the text-to-video retrieval task on MSRVTT and LSMDC benchmarks where our model outperforms all previous solutions by a large margin. Moreover, state-of-the-art results are achieved using a single model and without finetuning. This multidomain generalisation is achieved by a proper combination of different video caption datasets. We show that our practical approach for training on different datasets can improve test results of each other. Additionally, we check intersection between many popular datasets and show that MSRVTT as well as ActivityNet contains a significant overlap between the test and the training parts. More details are available at https://github.com/papermsucode/mdmmt.
BibTeX
@inproceedings{dzabraev2021mdmmt,
  author = {Maksim Dzabraev and Maksim Kalashnikov and Stepan Komkov and Aleksandr Petiushko},
  title = {{MDMMT: Multidomain Multimodal Transformer for Video Retrieval}},
  booktitle = {Workshops @ CVPR},
  year = {2021},
  url = {https://openreview.net/forum?id=jaIC-O0IPwn},
}
Figure