Abstract
In this paper, we present GEM as a General Evaluation benchmark for Multimodal tasks. Different from existing datasets such as GLUE, SuperGLUE, XGLUE and XTREME that mainly focus on natural language tasks, GEM is a large-scale vision-language benchmark, which consists of GEM-I for image-language tasks and GEM-V for video-language tasks. Comparing with existing multimodal datasets such as MSCOCO and Flicker30K for image-language tasks, YouCook2 and MSR-VTT for video-language tasks, GEM is not only the largest vision-language dataset covering image-language tasks and video-language tasks at the same time, but also labeled in multiple languages. We also provide two baseline models for this benchmark. We will release the dataset, code and baseline models, aiming to advance the development of multilingual multimodal research.
Introduction
In the field of Natural Language Processing (NLP), benchmarks like GLUE and XTREME have been instrumental in standardizing performance metrics across diverse tasks and languages. However, the multimodal domain—where models must simultaneously interpret visual and textual data—has lacked a comparable comprehensive standard. Existing datasets often suffer from language bias (primarily English-only), domain silos (focusing on either images or videos), and an over-reliance on "clean," human-written captions that do not reflect real-world usage.
GEM (General Evaluation benchmark for Multimodal tasks) addresses these gaps by providing a large-scale, multilingual benchmark designed to evaluate vision-language models across both image and video domains. Featuring over 1.2 million real-world data triplets across up to 30 languages, GEM tests the generalization capabilities of AI in complex, noisy environments.
The GEM Benchmark
GEM is divided into two primary sub-benchmarks: GEM-I (Image-Language) and GEM-V (Video-Language).
- GEM-I: Contains 1.2 million
{Query, Image, Title}triplets in 20 languages. - GEM-V: Contains 99,000
{Query, Video, Title}triplets in 30 languages.
Unlike traditional academic datasets that use descriptive sentences (e.g., "A brown dog sits on the grass"), GEM utilizes real-world search queries. For instance, a query might be "ruby tuesday steak and shrimp," requiring the model to bridge the gap between keyword-driven human intent and visual content.
Methodology and Baselines
To establish performance baselines, the researchers adapted two state-of-the-art architectures:
- M3P: A multilingual multimodal pre-trained model for GEM-I, utilizing a BERT-based encoder-decoder architecture.
- m-UniVL: A multilingual extension of the UniVL model for GEM-V. It employs XLM-R to handle 30 languages and utilizes both "loose" (single-modal) and "tight" (cross-modal) coupling to learn interactions between text and video.
A significant innovation in GEM is the inclusion of Side-Information. Models are provided with the Title of the web page where the visual media was hosted, serving as an auxiliary signal to improve alignment between the query and the visual data.
Key Insights and Results
The Impact of Metadata
The inclusion of web page titles as side-information led to dramatic performance improvements. In GEM-I, text-to-image retrieval Mean-Recall jumped from 19.85% (visual only) to 79.74% with titles. Similarly, in GEM-V, English video retrieval rose from 23.27% to 71.83%. This suggests that metadata is currently a vital "crutch" for AI when visual-textual alignment is challenging.
Multilingual Synergy and Domain Gaps
Experimental results show that fine-tuning models on all languages simultaneously generally outperforms language-specific training, confirming that multimodal models can effectively transfer knowledge across languages. However, a significant "domain gap" remains; models pre-trained on formal corpora like Wikipedia struggle with the informal, keyword-heavy nature of web search queries in zero-shot settings.
Cross-Modal Interaction
"Tightly coupled" models, which allow for deep interaction between text and video features, consistently outperformed loosely coupled versions. This highlights the necessity of deep cross-modal attention for sophisticated video understanding.
Conclusion
GEM represents a significant step toward robust, real-world multimodal AI. By moving beyond English-centric, descriptive datasets toward multilingual, query-based benchmarks, GEM provides a more accurate reflection of how AI interacts with the web. Future work will focus on improving "pure" visual retrieval performance and developing metrics that better capture the nuances of keyword-based captioning.
Summary updated