CMMU: A Benchmark for Chinese Multi-modal Multi-type Question Understanding and Reasoning

Zheqi He, Xinya Wu, Pengfei Zhou, Richeng Xuan, Guang Liu, Xi Yang, Qiannan Zhu, Hua Huang

Abstract

Multi-modal large language models (MLLMs) have demonstrated impressive capabilities in general knowledge comprehension. However, their performance in domain-specific reasoning remains a significant hurdle. Existing benchmarks often suffer from language bias (predominantly English) and format limitations (mostly multiple-choice). We introduce CMMU, a novel benchmark for multi-modal and multi-type question understanding in Chinese. Spanning seven subjects from primary to high school, CMMU challenges models with multiple-choice, multiple-response, and fill-in-the-blank questions. Our evaluation of state-of-the-art MLLMs reveals a substantial performance gap, particularly in reasoning-heavy subjects and non-standard question formats.

Introduction

The rapid evolution of Multi-modal Large Language Models (MLLMs) has necessitated more rigorous evaluation frameworks. While benchmarks like ScienceQA and MMMU have been instrumental, they are largely English-centric and rely heavily on multiple-choice questions (MCQs). This reliance on MCQs can lead to inflated performance metrics due to random guessing or the exploitation of positional biases. Furthermore, the lack of diverse question types, such as multiple-response (MRQ) and fill-in-the-blank (FBQ), limits the assessment of a model's true generative and reasoning capabilities.

CMMU addresses these limitations by providing a comprehensive Chinese-language dataset that incorporates diverse question formats and covers a wide range of academic subjects, from primary to high school levels.

The CMMU Benchmark

CMMU consists of 3,603 questions extracted from Chinese primary, middle, and high school examinations. The dataset covers seven core subjects: Math, Biology, Physics, Chemistry, Geography, Politics, and History.

Distribution of subjects and educational levels in the CMMU dataset

As shown in the distribution above, the benchmark maintains a balanced representation across subjects, with Math being the most prominent. The inclusion of three distinct question types—Multiple-Choice (MCQ), Multiple-Response (MRQ), and Fill-in-the-Blank (FBQ)—ensures a more holistic evaluation of model intelligence than traditional benchmarks.

Examples of multi-modal questions in CMMU across different subjects and types

Over 70% of the questions in CMMU include detailed answer explanations, providing a rich resource for analyzing model reasoning paths and identifying specific points of failure.

Evaluation Methodology

To ensure a robust evaluation, the authors implemented several innovative techniques designed to expose model weaknesses:

  1. CircularEval: To mitigate the impact of guessing in MCQs, the positions of options are cyclically shifted (e.g., ABCD becomes DABC). A model is only considered correct if it identifies the right answer across all possible permutations.
  2. Positional Error Variance (BiasRate): This metric quantifies a model's positional bias. By analyzing the distribution of incorrect answers across shifted options, the authors calculate the variance in the probability of choosing each position. A higher BiasRate indicates a stronger, content-independent preference for certain option letters.
  3. LLM-as-a-Judge: For the open-ended nature of FBQs, GPT-4 is utilized as an expert evaluator to determine the correctness of model-generated answers based on strict prompt templates.

Results and Key Insights

The evaluation of ten prominent MLLMs, including GPT-4V, Gemini-Pro, and various open-source models, yielded several critical insights:

  • The Performance Gap: GPT-4V emerged as the leader with an accuracy of 30.91%. However, this relatively low score underscores the extreme difficulty of the CMMU benchmark compared to general-purpose datasets.
  • Format Sensitivity: Models performed significantly worse on MRQs and FBQs compared to standard MCQs. This confirms that MCQ-only benchmarks likely overestimate model intelligence by allowing for successful guessing.
  • Subject Bias: Performance was notably higher in subjects requiring rote knowledge (e.g., History, Politics) than in those requiring complex spatial or mathematical reasoning (e.g., Physics, Math).
  • Positional Bias Trends: Interestingly, most models exhibited an "Anti-D" bias, rarely choosing option "D" as a default guess. More advanced models like GPT-4V showed lower overall BiasRate, indicating more robust reasoning and less reliance on formatting artifacts.
  • Error Analysis: A case study on GPT-4V's failures revealed that misleading reasoning (35%), image misunderstanding (27%), and question misunderstanding (13%) were the primary drivers of error.

Conclusion

CMMU provides a rigorous new standard for evaluating MLLMs in the Chinese language context. By moving beyond simple multiple-choice formats and introducing quantitative measures for positional bias, it exposes the significant limitations of current state-of-the-art models in domain-specific reasoning. Future work will focus on expanding the dataset size and further diversifying question formats to continue pushing the boundaries of multi-modal understanding.

Summary updated

Abstract

Multi-modal large language models(MLLMs) have achieved remarkable progress and demonstrated powerful knowledge comprehension and reasoning abilities. However, the mastery of domain-specific knowledge, which is essential for evaluating the intelligence of MLLMs, continues to be a challenge. Current multi-modal benchmarks for domain-specific knowledge concentrate on multiple-choice questions and are predominantly available in English, which imposes limitations on the comprehensiveness of the evaluation. To this end, we introduce CMMU, a novel benchmark for multi-modal and multi-type question understanding and reasoning in Chinese. CMMU consists of 3,603 questions in 7 subjects, covering knowledge from primary to high school. The questions can be categorized into 3 types: multiple-choice, multiple-response, and fill-in-the-blank, bringing greater challenges to MLLMs. In addition, we propose an evaluation strategy called Positional Error Variance for assessing multiple-choice questions. The strategy aims to perform a quantitative analysis of position bias. We evaluate seven open-source MLLMs along with GPT4-V, Gemini-Pro, and Qwen-VL-Plus. The results demonstrate that CMMU poses a significant challenge to the recent MLLMs. The data and code are available at https://github.com/FlagOpen/CMMU.
BibTeX
@inproceedings{he2024cmmu,
  author = {Zheqi He and Xinya Wu and Pengfei Zhou and Richeng Xuan and Guang Liu and Xi Yang and Qiannan Zhu and Hua Huang},
  title = {{CMMU: A Benchmark for Chinese Multi-modal Multi-type Question Understanding and Reasoning}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=5dkyA7giXO},
}
Figure