TL;DR
The paper introduces JMMMU (Japanese MMMU), the first large-scale benchmark evaluating Large Multimodal Models (LMMs) on expert-level, multi-disciplinary tasks in Japanese. By splitting the benchmark into culture-agnostic (translated) and culture-specific (newly crafted) subsets, it reveals critical gaps in both language translation robustness and deep cultural understanding among state-of-the-art models.
What Problem Does This Paper Address?
While Large Multimodal Models (LMMs) excel at English-centric benchmarks, their performance in other languages and cultural contexts remains under-evaluated. Existing Japanese vision-language benchmarks are typically small, focus primarily on common-sense knowledge, or rely entirely on direct translations of English datasets.
This creates two major issues: it fails to test advanced, college-level reasoning in non-English languages, and it completely overlooks whether a model actually understands the unique cultural context of a region (e.g., local history, art, and heritage) or is merely translating words on a surface level.
Methods
The authors developed JMMMU by separating evaluation into two distinct paradigms:
- Culture-Agnostic (CA) Evaluation: The authors selected and translated 720 expert-level questions from the well-known English MMMU benchmark into Japanese. Crucially, this involved not just text translation, but manual image editing to overwrite English text within diagrams and charts into Japanese, ensuring a pure language-variation test.
- Culture-Specific (CS) Evaluation: The team created 600 entirely new, college-level questions tailored to the Japanese educational and cultural context across fields like Japanese Art, History, and Heritage.
They evaluated 16 open-source LMMs and 3 leading proprietary models (GPT-4o, Gemini 1.5 Pro, Claude 3.5 Sonnet) using a robust rule-based parsing mechanism to evaluate accuracy regardless of minor formatting variations.
Datasets
The paper introduces JMMMU, a dataset containing 1,320 questions and 1,118 images across 28 subjects. It expands existing culture-aware Japanese benchmarks by over 10 times.
- Culture-Agnostic Subset (720 questions): Sourced and translated from MMMU's validation set across 24 subjects (e.g., Math, Physics, Biology, Business).
- Culture-Specific Subset (600 questions): Brand-new questions covering Japanese Art, Japanese Heritage, Japanese History, and World History (aligned with Japanese textbook curricula).
Key Insights and Ideas
- The "Translation Tax": LMMs suffer an average performance drop of 4.3% on culture-agnostic questions simply by changing the language from English to Japanese, even though the underlying logic and imagery are identical.
- Shallow vs. Deep Multilingualism: Evaluating models solely on translated benchmarks risks overestimating their capabilities. Some models perform well on translated Western concepts (CA) but fail drastically on local cultural knowledge (CS), exposing a lack of deep cultural grounding.
- Proprietary Divergence: Models that look identical on English benchmarks can diverge sharply when cultural context is introduced. For instance, GPT-4o and Claude 3.5 Sonnet perform nearly identically on English MMMU and Japanese CA, but GPT-4o outperforms Claude by a massive 15.7% on the culture-specific subset.
- Knowledge, Not Vision, is the Bottleneck: Error analysis on the culture-specific subset showed that over 53% of errors were due to a literal "Lack of Knowledge" (the model recognized the image correctly but didn't know the historical/cultural facts about it), whereas image recognition errors accounted for only ~31%.
Key Results
- Proprietary Models Lead: GPT-4o achieved the highest overall score at 58.6% (66.7% on Culture-Specific, 51.8% on Culture-Agnostic).
- Open-Source Gap: The best open-source model, LLaVA-OV-7B, reached 40.5% overall, highlighting a significant performance gap between open and closed models on expert tasks.
- Value of Local Fine-Tuning: Japanese-native open-source models (like LLaVA CALM2 and EvoVLM JP v2) demonstrated much higher relative performance on the Culture-Specific subset compared to general open-source models of similar size, proving that targeted training on native text successfully injects cultural competence.
Limitations and Future Work
- Subject Scope: JMMMU currently covers 28 subjects but needs expansion to include more diverse fields of study as LMM capabilities grow.
- Instruction-Following Issues: In Japanese, models frequently struggle to follow formatting constraints (e.g., providing full reasoning chains instead of just the requested multiple-choice letter), which can paradoxically skew evaluation metrics.
- Geographic Isolation: The benchmark is strictly focused on Japanese culture; the authors note that creating similar high-standard, culture-specific benchmarks for other underrepresented languages and regions is a vital next step for inclusive AI development.
Summary updated