MDMMT: Multidomain Multimodal Transformer for Video Retrieval

Maksim Dzabraev, Maksim Kalashnikov, Stepan Komkov, Aleksandr Petiushko

Abstract

We present a new state-of-the-art on the text to video retrieval task on MSRVTT and LSMDC benchmarks where our model outperforms all previous solutions by a large margin. Moreover, state-of-the-art results are achieved with a single model on two datasets without finetuning. This multidomain generalisation is achieved by a proper combination of different video caption datasets. We show that training on different datasets can improve test results of each other. Additionally we check intersection between many popular datasets and found that MSRVTT has a significant overlap between the test and the train parts, and the same situation is observed for ActivityNet.

Introduction: The Challenges of Video Retrieval

Text-to-video retrieval—finding the most relevant video clip based on a natural language query—is a cornerstone of modern search. However, the field faces three major hurdles:

  1. Specialization vs. Generalization: Most models are "overfitted" to a single dataset (like MSRVTT). A model that performs well on movie clips (LSMDC) often fails on YouTube-style content (ActivityNet).
  2. Data Scarcity: Individual video-caption datasets are relatively small, making it difficult for neural networks to learn the vast variety of real-world objects and actions.
  3. Dataset Leakage: The authors discovered a "dirty secret" in the community: many popular benchmarks have significant overlaps where nearly identical video segments appear in both the training and testing sets, artificially inflating performance scores.

Methodology: The Multimodal Transformer

The authors build upon the Multi-modal Transformer (MMT) framework, utilizing a two-stream architecture. In this setup, video and text are processed by separate networks into a shared embedding space. This is highly scalable because video embeddings can be pre-computed, requiring only a single forward pass of the text query at search time.

Comparison of single-stream and two-stream (dual-encoder) architectures for video retrieval.

Key technical contributions include:

  • Stronger Experts: Instead of raw pixels, the model uses pre-trained "experts" to extract features for Motion, RGB, and Audio. They found that CLIP (for images) and irCSN152 (trained on IG65M for motion) provided the strongest foundation.
  • Architectural Scaling: They increased the depth and width of the transformer encoders (moving from 4 layers to 9 layers) and applied more aggressive dropout (0.2) to prevent overfitting.
  • Weighted Multidomain Training: Rather than training on one dataset, they combined eight different datasets. They used a manual weighting strategy to sample from these datasets during training, ensuring the model didn't ignore smaller datasets or overfit on larger, noisier ones.
  • Two-Stage Data Cleaning: To ensure fair evaluation, they used YouTube IDs and embedding similarity to find and remove "near-duplicate" videos from the training sets that were already present in the test sets.

Datasets

The model was trained and evaluated on a massive "MALVYMTS" combination including core benchmarks like MSRVTT, ActivityNet, and LSMDC, alongside supplementary data from TwitterVines, YouCook2, MSVD, TGIF, and Something-something V2. The model also leverages pre-training on the HowTo100M dataset.

Results and Insights

Traditionally, researchers thought you needed to fine-tune a model for each specific dataset. MDMMT proves that a single model trained on a diverse "mega-dataset" can outperform specialized models on all fronts. The authors argue that the motion expert is the most critical component; if the expert fails to capture an action, the transformer cannot "recover" that information later.

Evaluation of multidomain training showing that combining datasets (MALVYMTS) improves Recall@5 across MSRVTT, ActivityNet, and LSMDC benchmarks compared to single-domain training.

MDMMT set new benchmarks across the board (measured by Recall@5):

  • MSRVTT (Full Clean): Achieved 49.5%, a massive jump over the previous MMT baseline.
  • LSMDC: Achieved 38.5%, outperforming the previous state-of-the-art by 8.6%.
  • ActivityNet: Achieved 45.1%, a 22.6% improvement over the standard MMT approach.

Notably, these results were achieved using the same model weights for all three benchmarks, demonstrating true multidomain capability. Furthermore, the discovery that ~10% of the MSRVTT test set overlaps with its training set highlights the importance of the authors' "MSRVTT full clean" split for honest evaluation.

Conclusion and Future Work

While MDMMT achieves state-of-the-art results, some limitations remain. The weights for combining different datasets were adjusted manually; automated "curriculum learning" could find more optimal sampling strategies. Additionally, training on the combined "mega-dataset" requires significant compute. Future work could involve "end-to-end" training where the video backbone is updated alongside the transformer, though this is currently computationally prohibitive for such large-scale data.

Summary updated

Abstract

We present a new state-of-the-art on the text to video retrieval task on MSRVTT and LSMDC benchmarks where our model outperforms all previous solutions by a large margin. Moreover, state-of-the-art results are achieved with a single model on two datasets without finetuning. This multidomain generalisation is achieved by a proper combination of different video caption datasets. We show that training on different datasets can improve test results of each other. Additionally we check intersection between many popular datasets and found that MSRVTT has a significant overlap between the test and the train parts, and the same situation is observed for ActivityNet.
BibTeX
@inproceedings{dzabraev2021mdmmt,
  author = {Maksim Dzabraev and Maksim Kalashnikov and Stepan Komkov and Aleksandr Petiushko},
  title = {{MDMMT: Multidomain Multimodal Transformer for Video Retrieval}},
  booktitle = {Workshops @ CVPR},
  year = {2021},
  url = {https://openreview.net/forum?id=jaIC-O0IPwn},
}
Figure