M3GIA: A Cognition Inspired Multilingual and Multimodal General Intelligence Ability Benchmark

Wei Song, Yadong Li, Xu Jhua, Guowei Wu, Lingfeng Ming, Kexin Yi, Weihua Luo, Yi Du, Fangda Guo, Kaicheng YuSubmission to ICLR 2025 · Withdrawn

Here is a concise, engaging summary of the paper.

M3GIA: Testing MLLMs Like Humans with a Cognition-Inspired “IQ Test”

TL;DR

Existing benchmarks evaluate Multimodal Large Language Models (MLLMs) on narrow, task-specific outputs rather than fundamental intelligence. To bridge this gap, the authors introduce M3GIA, the first multilingual, multimodal "IQ test" grounded in the well-established Cattell-Horn-Carroll (CHC) cognitive science framework. Evaluating 24 leading MLLMs against 480 human participants reveals that even cutting-edge models (like GPT-4o and Gemini-1.5-Pro) only barely scrape the lower boundary of human performance in English—and trail substantially behind in other languages.

What Problem Does This Paper Address?

Most current MLLM benchmarks (e.g., MMBench, MME, MM-Vet) test performance on arbitrary, task-oriented skills—like identifying object attributes or parsing charts. Because these benchmarks lack a solid theoretical foundation in cognitive psychology, high accuracy often reflects task memorization or superficial heuristic matching rather than actual general intelligence.

Furthermore, existing assessments are heavily English-centric and fail to disentangle whether a model's struggles stem from poor cognitive reasoning or limited non-English pre-training data.

Methods

The authors map MLLM capabilities to the Cattell-Horn-Carroll (CHC) theory of cognitive abilities, widely considered the most empirically validated structural model of human cognition:

  • Five Stratum-II Cognitive Factors:
  • Fluid Reasoning ($G_f$): Novel problem-solving (further split into Inductive, Deductive, and Quantitative reasoning).
  • Comprehension-Knowledge ($G_c$): Acquired culture-bound knowledge and vocabulary.
  • Quantitative Knowledge ($G_q$): Mathematical store and manipulation.
  • Visual Processing ($G_v$): Spatial rotation, pattern completion, and visual perception.
  • Reading and Writing ($G_{rw}$): Text-and-diagram literacy and passage comprehension.
  • Factor Decomposition & CFA Metric: Across these factors, the benchmark defines 5 clusters broken down into 18 narrow subtasks (e.g., Raven’s progressive matrices, concept formation, 3D spatial rotation).
  • General Intelligence Ability (GIA) Score: Instead of raw average accuracy, the authors employ a Confirmatory Factor Analysis (CFA) model fitted on human test data to compute a normalized, psychometrically sound GIA score.

Datasets

  • M3GIA Dataset: Contains 1,800 multiple-choice questions (1,200 VQA items), with 300 questions per language across six languages: English, Chinese, French, Spanish, Portuguese, and Korean.
  • 73% newly created by professionals from scratch (modeled after tests like the Woodcock-Johnson IV and Raven's Matrices) to prevent pre-training contamination/data leakage.
  • Sourced cultural content natively rather than translating from English to prevent English-centric cultural bias.
  • Human Baseline Data: Collected from 480 human participants (80 native speakers per language), providing a reliable baseline for normal human cognitive performance.

Key Insights and Ideas

  • Crystallized vs. Fluid Intelligence Gap: MLLMs excel at crystallized knowledge ($G_c$) and language-heavy tasks ($G_{rw}$) due to massive text pre-training, but lag severely in fluid reasoning ($G_f$) and visual-spatial processing ($G_v$).
  • Evidence of a Machine "$g$-Factor": The authors observed a clear "Winner Takes All" pattern: models that excelled in one cognitive domain consistently dominated across others. This mirrors the positive correlation of cognitive abilities seen in humans (Charles Spearman’s classic general factor, $g$).
  • Alignment with Real-World Human Preference: M3GIA scores achieved an $R^2 = 0.74$ linear correlation with Chatbot Arena (Vision) human Elo ratings—drastically outperforming traditional benchmarks like MMMU ($R^2 = 0.53$), MM-Vet ($R^2 = 0.55$), and MMBench ($R^2 = 0.03$).
  • Vision Pre-Training Over Parameter Scale: For visual processing ($G_v$), the number of training samples seen by the visual encoder matters far more than its raw parameter size (e.g., a 303M ViT trained on 32B samples achieved 38.3% accuracy, outperforming a 632M ViT trained on the same data volume at 37.5%).

Key Results

  • Overall Accuracy vs. Human: Humans averaged 76.9% overall accuracy across tasks. Among 24 evaluated models:
  • Gemini-1.5-Pro scored highest at 62.4%.
  • GPT-4o reached 59.8%.
  • Open-source frontrunners: InternVL-chat-v1.2-plus scored 51.9%; Mini-Gemini-34B scored 48.2%.
  • Visual-Spatial Processing ($G_v$): Humans scored 81.1%, while the best model (Gemini-1.5-Pro) only reached 53.8%.
  • Multilingual Disparity: Normalized English GIA scores for GPT-4o hit 86.5% of the human mean, but dropped noticeably in Chinese (68.7%), French (63.3%), and Korean (72.1%).
  • Emergence Threshold: Ablations on the Qwen-1.5 suite (1.8B to 72B) showed cognitive ability gains plateauing between 7B and 14B, with a sharp inflection in general cognitive ability emerging between 13B and 34B.

Limitations and Future Work

  • Omission of Auditory Processing ($G_a$): Auditory processing was excluded because most current MLLMs do not natively handle audio, though the authors plan to incorporate it as multimodal audio-vision-language models mature.
  • Language Coverage: Due to the high cost and labor required to handcraft expert psychometric items natively, the current benchmark is limited to six high-resource languages. Extending M3GIA to lower-resource and culturally distinct languages remains future work.

Summary updated

Abstract

As recent multi-modal large language models (MLLMs) have shown formidable proficiency on various complex tasks, there has been increasing attention on debating whether these models could eventually mirror human intelligence. However, existing benchmarks mainly focus on evaluating solely on task performance, such as the accuracy of identifying the attribute of an object. Combining well-developed cognitive science to understand the intelligence of MLLMs beyond superficial achievements remains largely unexplored. To this end, we introduce the first cognitive-driven multi-lingual and multi-modal benchmark to evaluate the general intelligence ability of MLLMs, dubbed M3GIA. Specifically, we identify five key cognitive factors based on the well-recognized Cattell-Horn-Carroll (CHC) model of intelligence and propose a novel evaluation metric. In addition, since most MLLMs are trained to perform in different languages, we go beyond English to encompass other languages, including Chinese, French, Spanish, Portuguese and Korean, to construct our M3GIA. We make sure all the data relevant to the cultural backgrounds are collected from their native context to avoid English-centric bias. We collected a significant corpus of data from human participants, revealing that the most advanced MLLM barely reaches the lower boundary of human performance in English, and there remains a pronounced disparity in the other five languages. Importantly, we found that designing IQ tests for MLLMs is crucial, as the evaluation of M3GIA achieves a significantly stronger alignment with human preferences compared to traditional task-oriented benchmarks. Moreover, grounded in CHC theory, we discovered that the number of samples seen by the vision encoder has a greater influence on the model's visual capabilities than its parameter size.
BibTeX
@misc{song2024m,
  author = {Wei Song and Yadong Li and Xu Jhua and Guowei Wu and Lingfeng Ming and Kexin Yi and Weihua Luo and Yi Du and Fangda Guo and Kaicheng Yu},
  title = {{M3GIA: A Cognition Inspired Multilingual and Multimodal General Intelligence Ability Benchmark}},
  year = {2024},
  eprint = {2406.05343},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2406.05343},
}
Figure