JMMMU: A Japanese Massive Multi-discipline Multimodal Understanding Benchmark for Culture-aware Evaluation

Shota Onohara, Atsuyuki Miyai, Yuki Imajuku, Kazuki Egashira, Jeonghun Baek, Xiang Yue, Graham Neubig, Kiyoharu Aizawa

TL;DR

The paper introduces JMMMU (Japanese MMMU), the first large-scale benchmark evaluating Large Multimodal Models (LMMs) on expert-level, multi-disciplinary tasks in Japanese. By splitting the benchmark into culture-agnostic (translated) and culture-specific (newly crafted) subsets, it reveals critical gaps in both language translation robustness and deep cultural understanding among state-of-the-art models.

What Problem Does This Paper Address?

While Large Multimodal Models (LMMs) excel at English-centric benchmarks, their performance in other languages and cultural contexts remains under-evaluated. Existing Japanese vision-language benchmarks are typically small, focus primarily on common-sense knowledge, or rely entirely on direct translations of English datasets.

This creates two major issues: it fails to test advanced, college-level reasoning in non-English languages, and it completely overlooks whether a model actually understands the unique cultural context of a region (e.g., local history, art, and heritage) or is merely translating words on a surface level.

Methods

The authors developed JMMMU by separating evaluation into two distinct paradigms:

  1. Culture-Agnostic (CA) Evaluation: The authors selected and translated 720 expert-level questions from the well-known English MMMU benchmark into Japanese. Crucially, this involved not just text translation, but manual image editing to overwrite English text within diagrams and charts into Japanese, ensuring a pure language-variation test.
  2. Culture-Specific (CS) Evaluation: The team created 600 entirely new, college-level questions tailored to the Japanese educational and cultural context across fields like Japanese Art, History, and Heritage.

They evaluated 16 open-source LMMs and 3 leading proprietary models (GPT-4o, Gemini 1.5 Pro, Claude 3.5 Sonnet) using a robust rule-based parsing mechanism to evaluate accuracy regardless of minor formatting variations.

Datasets

The paper introduces JMMMU, a dataset containing 1,320 questions and 1,118 images across 28 subjects. It expands existing culture-aware Japanese benchmarks by over 10 times.

  • Culture-Agnostic Subset (720 questions): Sourced and translated from MMMU's validation set across 24 subjects (e.g., Math, Physics, Biology, Business).
  • Culture-Specific Subset (600 questions): Brand-new questions covering Japanese Art, Japanese Heritage, Japanese History, and World History (aligned with Japanese textbook curricula).
Figure 1: Overview of the JMMMU dataset. JMMMU includes 720 culture-agnostic (translation-based) questions and 600 culture-specific (newly created) questions, totaling 1,320 questions, thus expanding the existing culture- aware Japanese benchmark (Inoue et al., 2024b) by over 10 times. JMMMU serves as a diagnostic tool for assessing both Japanese cultural understanding and culture-agnostic language understanding capability.

Key Insights and Ideas

  • The "Translation Tax": LMMs suffer an average performance drop of 4.3% on culture-agnostic questions simply by changing the language from English to Japanese, even though the underlying logic and imagery are identical.
  • Shallow vs. Deep Multilingualism: Evaluating models solely on translated benchmarks risks overestimating their capabilities. Some models perform well on translated Western concepts (CA) but fail drastically on local cultural knowledge (CS), exposing a lack of deep cultural grounding.
  • Proprietary Divergence: Models that look identical on English benchmarks can diverge sharply when cultural context is introduced. For instance, GPT-4o and Claude 3.5 Sonnet perform nearly identically on English MMMU and Japanese CA, but GPT-4o outperforms Claude by a massive 15.7% on the culture-specific subset.
  • Knowledge, Not Vision, is the Bottleneck: Error analysis on the culture-specific subset showed that over 53% of errors were due to a literal "Lack of Knowledge" (the model recognized the image correctly but didn't know the historical/cultural facts about it), whereas image recognition errors accounted for only ~31%.

Key Results

  • Proprietary Models Lead: GPT-4o achieved the highest overall score at 58.6% (66.7% on Culture-Specific, 51.8% on Culture-Agnostic).
  • Open-Source Gap: The best open-source model, LLaVA-OV-7B, reached 40.5% overall, highlighting a significant performance gap between open and closed models on expert tasks.
  • Value of Local Fine-Tuning: Japanese-native open-source models (like LLaVA CALM2 and EvoVLM JP v2) demonstrated much higher relative performance on the Culture-Specific subset compared to general open-source models of similar size, proving that targeted training on native text successfully injects cultural competence.

Limitations and Future Work

  • Subject Scope: JMMMU currently covers 28 subjects but needs expansion to include more diverse fields of study as LMM capabilities grow.
  • Instruction-Following Issues: In Japanese, models frequently struggle to follow formatting constraints (e.g., providing full reasoning chains instead of just the requested multiple-choice letter), which can paradoxically skew evaluation metrics.
  • Geographic Isolation: The benchmark is strictly focused on Japanese culture; the authors note that creating similar high-standard, culture-specific benchmarks for other underrepresented languages and regions is a vital next step for inclusive AI development.

Summary updated

Abstract

Accelerating research on Large Multimodal Models (LMMs) in non-English languages is crucial for enhancing user experiences across broader populations. In this paper, we introduce JMMMU (Japanese MMMU), the first large-scale Japanese benchmark designed to evaluate LMMs on expert-level tasks based on the Japanese cultural context. To facilitate comprehensive culture-aware evaluation, JMMMU features two complementary subsets: (i) culture-agnostic (CA) subset, where the culture-independent subjects (e.g., Math) are selected and translated into Japanese, enabling one-to-one comparison with its English counterpart MMMU; and (ii) culture-specific (CS) subset, comprising newly crafted subjects that reflect Japanese cultural context. Using the CA subset, we observe performance drop in many LMMs when evaluated in Japanese, which is purely attributable to language variation. Using the CS subset, we reveal their inadequate Japanese cultural understanding. Further, by combining both subsets, we identify that some LMMs perform well on the CA subset but not on the CS subset, exposing a shallow understanding of the Japanese language that lacks depth in cultural understanding. We hope this work will not only help advance LMM performance in Japanese but also serve as a guideline to create high-standard, culturally diverse benchmarks for multilingual LMM development. The project page is https://mmmu-japanese-benchmark.github.io/JMMMU/.
BibTeX
@inproceedings{onohara2025jmmmu,
  author = {Shota Onohara and Atsuyuki Miyai and Yuki Imajuku and Kazuki Egashira and Jeonghun Baek and Xiang Yue and Graham Neubig and Kiyoharu Aizawa},
  title = {{JMMMU: A Japanese Massive Multi-discipline Multimodal Understanding Benchmark for Culture-aware Evaluation}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
  doi = {10.18653/v1/2025.naacl-long.43},
}
Figure