TL;DR
This paper introduces GEM, the first large-scale, multilingual benchmark for evaluating multimodal vision-language models across both image and video tasks. GEM aims to address the limitations of existing benchmarks that are primarily English-only and focus on single modalities, providing a more comprehensive and realistic evaluation for models intended for global applications.
What Problem Does This Paper Address?
The field of multimodal AI, which combines vision and language understanding, has seen rapid progress thanks to large-scale pre-trained models. However, evaluating these models has been hampered by a lack of comprehensive benchmarks. Existing benchmarks like GLUE and SuperGLUE are designed for natural language tasks, while vision-language benchmarks are often limited to English and specific tasks like image-text retrieval or captioning. This leaves a significant gap in evaluating how well these models generalize across different languages and modalities (images and videos) in real-world scenarios. The authors argue that current benchmarks don't adequately reflect the diversity and complexity of multimodal data encountered in multilingual contexts, hindering the development of truly robust and globally applicable AI systems.
Methods
The core contribution of this paper is the creation of the GEM benchmark, which is divided into two main parts:
- GEM-I: Focuses on image-language tasks, specifically text-to-image retrieval and image captioning.
- GEM-V: Focuses on video-language tasks, specifically text-to-video retrieval and video captioning.
A key innovation of GEM is its multilingual nature. The natural language contexts (queries and titles) are collected from a commercial search engine, ensuring a diverse range of languages and real-world query styles. GEM-I includes data in 20 languages, while GEM-V supports 30 languages.
To demonstrate the benchmark's utility, the authors also provide two baseline models:
- M³P: A state-of-the-art multilingual multimodal pre-trained model for image-language tasks.
- m-UniVL: A multilingual extension of the UniVL model, adapted for video-language tasks.
These models are fine-tuned on the GEM datasets to establish initial performance benchmarks.
Datasets
The paper introduces two new benchmark datasets:
- GEM-I: Contains approximately 1.2 million {Query, Image, Title} triplets across 20 languages for text-to-image retrieval and image captioning.
- GEM-V: Contains approximately 99,000 {Query, Video, Title} triplets across 30 languages for text-to-video retrieval and video captioning.
The data for both GEM-I and GEM-V is sourced from a commercial search engine, with titles of web pages serving as auxiliary information. This approach contrasts with existing datasets like MSCOCO and Flicker30K (image tasks) and YouCook2 and MSR-VTT (video tasks) by offering a larger scale, multilingual support, and a focus on more natural, less grammatically perfect queries.
Key Insights and Ideas
- Multilingualism is Crucial: The success of multimodal models in real-world applications hinges on their ability to perform across diverse languages, not just English. GEM highlights this by providing a benchmark that directly tests this capability.
- Real-World Queries are Messy: GEM's use of queries from a commercial search engine reveals that natural language inputs are often shorter, less grammatically structured, and more diverse than those found in curated academic datasets. This presents a more realistic challenge for models.
- Title Information is Valuable: The inclusion of web page titles as side information significantly boosts performance in retrieval tasks, underscoring the importance of contextual metadata in multimodal understanding.
- Fine-tuning on All Data is Beneficial: The results suggest that fine-tuning models on data from all available languages in GEM leads to better generalization, especially for languages with limited training data, demonstrating the power of cross-lingual transfer.
Key Results
- GEM-I Text-to-Image Retrieval:
- Zero-shot performance of M³P was poor, indicating the need for fine-tuning on domain-specific data.
- Fine-tuning M³P on all GEM-I data significantly improved performance, with the model achieving mean-recall scores up to 93.75% (Q→I+T, fine-tuned on all).
- Including title information (Q→I+T) consistently improved retrieval performance over using only the image (Q→I).
- GEM-I Image Captioning:
- M³P showed relatively poor performance in image captioning, attributed to the short, keyword-like nature of GEM queries differing from pre-training data.
- The dataset demonstrated its ability to evaluate multilingual multimodal captioning, with scores like ROUGE-L reaching up to 20.18 on average across languages.
- GEM-V Text-to-Video Retrieval:
- The "tight" coupling model (m-UniVL(tight)) outperformed the "loose" model, highlighting the benefit of cross-modal interaction.
- Including title information (Q→V+T) provided a substantial performance gain, with mean-recall scores reaching up to 74.47% for m-UniVL(tight) (fine-tuned on ALL).
- Fine-tuning on all languages generally yielded better results than fine-tuning on individual languages.
- GEM-V Video Captioning:
- Performance was lower when relying solely on video (V→Q), emphasizing the importance of title information for captioning.
- Using both video and title (V+T→Q) achieved significantly better results, with average ROUGE-L scores reaching up to 36.73.
Limitations and Future Work
- Data Quality: While efforts were made to filter data, the authors acknowledge that some noise might still exist in the collected queries and titles.
- Image Captioning Performance: The performance on image captioning tasks was noted as relatively poor, suggesting that current models may struggle with the naturalistic, less structured queries present in GEM.
- Human Face Handling in Videos: For GEM-V, specific measures were not taken to address human faces in videos, as the distribution method is via URLs, and modifying videos could violate copyright.
- Future Work: The authors hope GEM will spur further research in multilingual multimodal learning. They plan to release the dataset, code, and baseline models to facilitate this. Future work could involve developing more robust models that can better handle the nuances of natural language queries and improve performance on tasks like image captioning.
Summary updated