Abstract
Multi-modal large language models (MLLMs) have achieved remarkable progress and demonstrated powerful knowledge comprehension and reasoning abilities. However, the mastery of domain-specific knowledge, which is essential for evaluating the intelligence of MLLMs, continues to be a challenge. Current multi-modal benchmarks for domain-specific knowledge concentrate on multiple-choice questions and are predominantly available in English, which imposes limitations on the comprehensiveness of the evaluation. To this end, we introduce CMMU, a novel benchmark for multi-modal and multi-type question understanding and reasoning in Chinese. CMMU consists of 3,603 questions in 7 subjects, covering knowledge from primary to high school. The questions can be categorized into 3 types: multiple-choice, multiple-response, and fill-in-the-blank, bringing greater challenges to MLLMs. In addition, we propose an evaluation strategy called Positional Error Variance for assessing multiple-choice questions. The strategy aims to perform a quantitative analysis of position bias. We evaluate seven open-source MLLMs along with GPT4-V, Gemini-Pro, and Qwen-VL-Plus. The results demonstrate that CMMU poses a significant challenge to the recent MLLMs.
Introduction: The Need for Rigorous Domain Evaluation
While Multi-modal Large Language Models (MLLMs) have made massive strides in general image description and common-sense reasoning, evaluating their intelligence in specialized domains remains difficult. Most existing benchmarks suffer from language bias (predominantly English), format limitations (relying almost exclusively on multiple-choice questions), and superficiality. Many benchmarks test simple perception rather than the deep reasoning required to understand how components in a technical diagram interact.
To bridge this gap, the authors introduce CMMU (Chinese Multi-modal Multi-type), a dataset that requires models to integrate complex Chinese text with technical diagrams to solve problems ranging from primary to high school levels.
The CMMU Dataset
The CMMU benchmark is a comprehensive collection of 3,603 questions (split into 1,800 validation and 1,803 test items). It spans seven core subjects: Math, Biology, Physics, Chemistry, Geography, Politics, and History.
Unlike standard benchmarks, CMMU is the first to combine the Chinese language with multi-modal inputs and multi-type questions, including Multiple-Choice (MCQ), Multiple-Response (MRQ), and Fill-in-the-Blank (FBQ). This variety prevents models from relying on simple elimination strategies and forces true reasoning.
Methodology and Evaluation Strategy
The researchers implemented a sophisticated evaluation framework to ensure the robustness of the results:
- Multi-type Evaluation: For FBQs, GPT-4 is used as a "judge" to determine if a model's generated text is semantically equivalent to the ground truth.
- CircularEval: To ensure a model actually knows the answer rather than getting lucky, the "circular shift" method rotates options (e.g., ABCD becomes BCDA). A model must pick the correct answer in every permutation to earn a point.
- Positional Error Variance: This new metric quantifies position bias, measuring whether a model has a "favorite" option (like always picking 'C' when unsure).
Key Insights and Results
The evaluation of 11 prominent models, including GPT-4V, Gemini-Pro, and Qwen-VL-Plus, yielded several key insights:
- The "Guessing" Penalty: The inclusion of MRQs and FBQs significantly dropped model performance. In multiple-choice, a model has a 25% floor for guessing; in a fill-in-the-blank math problem, that floor is effectively zero.
- Subject Sensitivity: Models performed much better on "knowledge-heavy" subjects like Politics and History (~45-50% for GPT-4V) compared to "reasoning-heavy" subjects like Math and Physics (~13-28%). This suggests MLLMs are currently better at retrieving facts than performing multi-step logical or spatial reasoning.
- The Difficulty Gap: There is a massive performance cliff between "Normal" and "Hard" questions. On the "Hard" set for multi-response questions, seven out of eleven models scored less than 10% accuracy.
Overall, GPT-4V emerged as the leader with 30.91% accuracy, followed by Qwen-VL-Plus at 27.73%. Open-source models like LLaVA or CogVLM lagged significantly, hovering between 5% and 18%.
Conclusion
CMMU reveals that even state-of-the-art MLLMs still struggle with domain-specific reasoning and inherent positional biases. While 3,600 questions provide a solid foundation, future work will focus on increasing the dataset's scale and subject balance, particularly at the primary school level, to continue pushing the boundaries of multi-modal understanding.
Summary updated