Abstract
Multi-modal large language models (MLLMs) have demonstrated impressive capabilities in general knowledge comprehension. However, their performance in domain-specific reasoning remains a significant hurdle. Existing benchmarks often suffer from language bias (predominantly English) and format limitations (mostly multiple-choice). We introduce CMMU, a novel benchmark for multi-modal and multi-type question understanding in Chinese. Spanning seven subjects from primary to high school, CMMU challenges models with multiple-choice, multiple-response, and fill-in-the-blank questions. Our evaluation of state-of-the-art MLLMs reveals a substantial performance gap, particularly in reasoning-heavy subjects and non-standard question formats.
Introduction
The rapid evolution of Multi-modal Large Language Models (MLLMs) has necessitated more rigorous evaluation frameworks. While benchmarks like ScienceQA and MMMU have been instrumental, they are largely English-centric and rely heavily on multiple-choice questions (MCQs). This reliance on MCQs can lead to inflated performance metrics due to random guessing or the exploitation of positional biases. Furthermore, the lack of diverse question types, such as multiple-response (MRQ) and fill-in-the-blank (FBQ), limits the assessment of a model's true generative and reasoning capabilities.
CMMU addresses these limitations by providing a comprehensive Chinese-language dataset that incorporates diverse question formats and covers a wide range of academic subjects, from primary to high school levels.
The CMMU Benchmark
CMMU consists of 3,603 questions extracted from Chinese primary, middle, and high school examinations. The dataset covers seven core subjects: Math, Biology, Physics, Chemistry, Geography, Politics, and History.
As shown in the distribution above, the benchmark maintains a balanced representation across subjects, with Math being the most prominent. The inclusion of three distinct question types—Multiple-Choice (MCQ), Multiple-Response (MRQ), and Fill-in-the-Blank (FBQ)—ensures a more holistic evaluation of model intelligence than traditional benchmarks.
Over 70% of the questions in CMMU include detailed answer explanations, providing a rich resource for analyzing model reasoning paths and identifying specific points of failure.
Evaluation Methodology
To ensure a robust evaluation, the authors implemented several innovative techniques designed to expose model weaknesses:
- CircularEval: To mitigate the impact of guessing in MCQs, the positions of options are cyclically shifted (e.g., ABCD becomes DABC). A model is only considered correct if it identifies the right answer across all possible permutations.
- Positional Error Variance (BiasRate): This metric quantifies a model's positional bias. By analyzing the distribution of incorrect answers across shifted options, the authors calculate the variance in the probability of choosing each position. A higher
BiasRateindicates a stronger, content-independent preference for certain option letters. - LLM-as-a-Judge: For the open-ended nature of FBQs, GPT-4 is utilized as an expert evaluator to determine the correctness of model-generated answers based on strict prompt templates.
Results and Key Insights
The evaluation of ten prominent MLLMs, including GPT-4V, Gemini-Pro, and various open-source models, yielded several critical insights:
- The Performance Gap: GPT-4V emerged as the leader with an accuracy of 30.91%. However, this relatively low score underscores the extreme difficulty of the CMMU benchmark compared to general-purpose datasets.
- Format Sensitivity: Models performed significantly worse on MRQs and FBQs compared to standard MCQs. This confirms that MCQ-only benchmarks likely overestimate model intelligence by allowing for successful guessing.
- Subject Bias: Performance was notably higher in subjects requiring rote knowledge (e.g., History, Politics) than in those requiring complex spatial or mathematical reasoning (e.g., Physics, Math).
- Positional Bias Trends: Interestingly, most models exhibited an "Anti-D" bias, rarely choosing option "D" as a default guess. More advanced models like GPT-4V showed lower overall
BiasRate, indicating more robust reasoning and less reliance on formatting artifacts. - Error Analysis: A case study on GPT-4V's failures revealed that misleading reasoning (35%), image misunderstanding (27%), and question misunderstanding (13%) were the primary drivers of error.
Conclusion
CMMU provides a rigorous new standard for evaluating MLLMs in the Chinese language context. By moving beyond simple multiple-choice formats and introducing quantitative measures for positional bias, it exposes the significant limitations of current state-of-the-art models in domain-specific reasoning. Future work will focus on expanding the dataset size and further diversifying question formats to continue pushing the boundaries of multi-modal understanding.
Summary updated