Abstract
Recent advancements in large vision-language models (VLMs) have primarily focused on English, with limited attention given to other languages. To address this gap, we introduce MEENA (also known as PersianMMMU), the first dataset designed to evaluate Persian VLMs across scientific, reasoning, and human-level understanding tasks. Our dataset comprises approximately 7,500 Persian and 3,000 English questions, covering a wide range of topics such as reasoning, mathematics, physics, diagrams, charts, and Persian art and literature. Key features of MEENA include: (1) diverse subject coverage spanning various educational levels, from primary to upper secondary school, (2) rich metadata, including difficulty levels and descriptive answers, (3) original Persian data that preserves cultural nuances, (4) a bilingual structure to assess cross-linguistic performance, and (5) a series of diverse experiments assessing various capabilities, including overall performance, the model's ability to attend to images, and its tendency to generate hallucinations. We hope this benchmark contributes to enhancing VLM capabilities beyond English.
Introduction: Addressing the Multimodal Language Gap
While Large Vision-Language Models (VLMs) like GPT-4o and Gemini have shown remarkable capabilities, their development and evaluation have been overwhelmingly English-centric. Existing benchmarks for non-English languages often rely on machine translations of English datasets, which can lead to "cultural misalignment" and the loss of linguistic nuances. For a language like Persian, previous benchmarks were limited to simple tasks like image captioning, lacking the depth required to test human-level understanding in specialized domains.
The MEENA Dataset
The core contribution of this work is the MEENA dataset, which consists of over 10,000 high-quality multiple-choice questions sourced from real-world Iranian educational exams. This includes 7,483 Persian questions and 3,067 high-fidelity English translations to facilitate cross-linguistic comparison.
MEENA spans primary to upper secondary school levels across 27 subtasks, including Mathematics, Natural Sciences, Social Sciences, Humanities, and a specialized subset for Persian Art. Unlike many datasets, MEENA includes rich metadata such as difficulty levels (1–5), descriptive answer explanations, and "trap" indicators for misleading choices.
Experimental Framework
The researchers developed a rigorous evaluation framework centered on five distinct experimental settings to isolate how models utilize visual information:
- Zero-Shot (ZS): The standard baseline.
- In-Context Learning (ICL): Providing four manually selected example pairs.
- First Describe (FD): Forcing the model to describe the image before answering, acting as a "visual chain-of-thought."
- Wrong Image (WI): Providing an intentionally mismatched image to test for hallucinations.
- Without Image (WO): Providing only text to determine if the question truly requires visual input.
Key Results and Insights
Evaluation of state-of-the-art models revealed a significant "language tax," with performance on Persian tasks consistently lower than on English counterparts. Models also struggled more with reasoning-based tasks compared to factual recall, showing a 10–19% performance gap.
As shown in the results, Gemini 2.0 Flash and GPT-4o emerged as the leaders. Gemini 2.0 Flash achieved the highest accuracy on the Persian MEENA dataset in the "First Describe" setting (0.504), suggesting that forcing explicit visual processing helps ground reasoning. Interestingly, the "Wrong Image" experiment highlighted that Gemini 2.0 Flash was significantly more robust than GPT-4o in detecting mismatched visual contexts, though both models still fall short of human performance levels.
Conclusion
MEENA establishes a vital foundation for developing VLMs that are culturally aware and capable of expert-level reasoning beyond English. While current top-tier models show promise, the sharp drop in accuracy as question difficulty increases indicates that significant work remains to achieve true human-level multimodal understanding in non-English languages.
Summary updated