CMMU: A Benchmark for Chinese Multi-modal Multi-type Question Understanding and Reasoning

Zheqi He, Xinya Wu, Pengfei Zhou, Richeng Xuan, Guang Liu, Xi Yang, Qiannan Zhu, Hua Huang

Abstract

Multi-modal large language models (MLLMs) have achieved remarkable progress and demonstrated powerful knowledge comprehension and reasoning abilities. However, the mastery of domain-specific knowledge, which is essential for evaluating the intelligence of MLLMs, continues to be a challenge. Current multi-modal benchmarks for domain-specific knowledge concentrate on multiple-choice questions and are predominantly available in English, which imposes limitations on the comprehensiveness of the evaluation. To this end, we introduce CMMU, a novel benchmark for multi-modal and multi-type question understanding and reasoning in Chinese. CMMU consists of 3,603 questions in 7 subjects, covering knowledge from primary to high school. The questions can be categorized into 3 types: multiple-choice, multiple-response, and fill-in-the-blank, bringing greater challenges to MLLMs. In addition, we propose an evaluation strategy called Positional Error Variance for assessing multiple-choice questions. The strategy aims to perform a quantitative analysis of position bias. We evaluate seven open-source MLLMs along with GPT4-V, Gemini-Pro, and Qwen-VL-Plus. The results demonstrate that CMMU poses a significant challenge to the recent MLLMs.

Introduction: The Need for Rigorous Domain Evaluation

While Multi-modal Large Language Models (MLLMs) have made massive strides in general image description and common-sense reasoning, evaluating their intelligence in specialized domains remains difficult. Most existing benchmarks suffer from language bias (predominantly English), format limitations (relying almost exclusively on multiple-choice questions), and superficiality. Many benchmarks test simple perception rather than the deep reasoning required to understand how components in a technical diagram interact.

A sample physics question from CMMU requiring complex reasoning about circuit components.

To bridge this gap, the authors introduce CMMU (Chinese Multi-modal Multi-type), a dataset that requires models to integrate complex Chinese text with technical diagrams to solve problems ranging from primary to high school levels.

The CMMU Dataset

The CMMU benchmark is a comprehensive collection of 3,603 questions (split into 1,800 validation and 1,803 test items). It spans seven core subjects: Math, Biology, Physics, Chemistry, Geography, Politics, and History.

Distribution of subjects and educational levels in the CMMU benchmark.

Unlike standard benchmarks, CMMU is the first to combine the Chinese language with multi-modal inputs and multi-type questions, including Multiple-Choice (MCQ), Multiple-Response (MRQ), and Fill-in-the-Blank (FBQ). This variety prevents models from relying on simple elimination strategies and forces true reasoning.

Methodology and Evaluation Strategy

The researchers implemented a sophisticated evaluation framework to ensure the robustness of the results:

  • Multi-type Evaluation: For FBQs, GPT-4 is used as a "judge" to determine if a model's generated text is semantically equivalent to the ground truth.
  • CircularEval: To ensure a model actually knows the answer rather than getting lucky, the "circular shift" method rotates options (e.g., ABCD becomes BCDA). A model must pick the correct answer in every permutation to earn a point.
  • Positional Error Variance: This new metric quantifies position bias, measuring whether a model has a "favorite" option (like always picking 'C' when unsure).

Key Insights and Results

The evaluation of 11 prominent models, including GPT-4V, Gemini-Pro, and Qwen-VL-Plus, yielded several key insights:

  • The "Guessing" Penalty: The inclusion of MRQs and FBQs significantly dropped model performance. In multiple-choice, a model has a 25% floor for guessing; in a fill-in-the-blank math problem, that floor is effectively zero.
  • Subject Sensitivity: Models performed much better on "knowledge-heavy" subjects like Politics and History (~45-50% for GPT-4V) compared to "reasoning-heavy" subjects like Math and Physics (~13-28%). This suggests MLLMs are currently better at retrieving facts than performing multi-step logical or spatial reasoning.
  • The Difficulty Gap: There is a massive performance cliff between "Normal" and "Hard" questions. On the "Hard" set for multi-response questions, seven out of eleven models scored less than 10% accuracy.

Overall, GPT-4V emerged as the leader with 30.91% accuracy, followed by Qwen-VL-Plus at 27.73%. Open-source models like LLaVA or CogVLM lagged significantly, hovering between 5% and 18%.

Conclusion

CMMU reveals that even state-of-the-art MLLMs still struggle with domain-specific reasoning and inherent positional biases. While 3,600 questions provide a solid foundation, future work will focus on increasing the dataset's scale and subject balance, particularly at the primary school level, to continue pushing the boundaries of multi-modal understanding.

Summary updated

Abstract

Multi-modal large language models(MLLMs) have achieved remarkable progress and demonstrated powerful knowledge comprehension and reasoning abilities. However, the mastery of domain-specific knowledge, which is essential for evaluating the intelligence of MLLMs, continues to be a challenge. Current multi-modal benchmarks for domain-specific knowledge concentrate on multiple-choice questions and are predominantly available in English, which imposes limitations on the comprehensiveness of the evaluation. To this end, we introduce CMMU, a novel benchmark for multi-modal and multi-type question understanding and reasoning in Chinese. CMMU consists of 3,603 questions in 7 subjects, covering knowledge from primary to high school. The questions can be categorized into 3 types: multiple-choice, multiple-response, and fill-in-the-blank, bringing greater challenges to MLLMs. In addition, we propose an evaluation strategy called Positional Error Variance for assessing multiple-choice questions. The strategy aims to perform a quantitative analysis of position bias. We evaluate seven open-source MLLMs along with GPT4-V, Gemini-Pro, and Qwen-VL-Plus. The results demonstrate that CMMU poses a significant challenge to the recent MLLMs. The data and code are available at https://github.com/FlagOpen/CMMU.
BibTeX
@inproceedings{he2024cmmu,
  author = {Zheqi He and Xinya Wu and Pengfei Zhou and Richeng Xuan and Guang Liu and Xi Yang and Qiannan Zhu and Hua Huang},
  title = {{CMMU: A Benchmark for Chinese Multi-modal Multi-type Question Understanding and Reasoning}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=5dkyA7giXO},
}
Figure