Here is a concise, engaging summary of the paper.
M3GIA: Testing MLLMs Like Humans with a Cognition-Inspired “IQ Test”
TL;DR
Existing benchmarks evaluate Multimodal Large Language Models (MLLMs) on narrow, task-specific outputs rather than fundamental intelligence. To bridge this gap, the authors introduce M3GIA, the first multilingual, multimodal "IQ test" grounded in the well-established Cattell-Horn-Carroll (CHC) cognitive science framework. Evaluating 24 leading MLLMs against 480 human participants reveals that even cutting-edge models (like GPT-4o and Gemini-1.5-Pro) only barely scrape the lower boundary of human performance in English—and trail substantially behind in other languages.
What Problem Does This Paper Address?
Most current MLLM benchmarks (e.g., MMBench, MME, MM-Vet) test performance on arbitrary, task-oriented skills—like identifying object attributes or parsing charts. Because these benchmarks lack a solid theoretical foundation in cognitive psychology, high accuracy often reflects task memorization or superficial heuristic matching rather than actual general intelligence.
Furthermore, existing assessments are heavily English-centric and fail to disentangle whether a model's struggles stem from poor cognitive reasoning or limited non-English pre-training data.
Methods
The authors map MLLM capabilities to the Cattell-Horn-Carroll (CHC) theory of cognitive abilities, widely considered the most empirically validated structural model of human cognition:
- Five Stratum-II Cognitive Factors:
- Fluid Reasoning ($G_f$): Novel problem-solving (further split into Inductive, Deductive, and Quantitative reasoning).
- Comprehension-Knowledge ($G_c$): Acquired culture-bound knowledge and vocabulary.
- Quantitative Knowledge ($G_q$): Mathematical store and manipulation.
- Visual Processing ($G_v$): Spatial rotation, pattern completion, and visual perception.
- Reading and Writing ($G_{rw}$): Text-and-diagram literacy and passage comprehension.
- Factor Decomposition & CFA Metric: Across these factors, the benchmark defines 5 clusters broken down into 18 narrow subtasks (e.g., Raven’s progressive matrices, concept formation, 3D spatial rotation).
- General Intelligence Ability (GIA) Score: Instead of raw average accuracy, the authors employ a Confirmatory Factor Analysis (CFA) model fitted on human test data to compute a normalized, psychometrically sound GIA score.
Datasets
- M3GIA Dataset: Contains 1,800 multiple-choice questions (1,200 VQA items), with 300 questions per language across six languages: English, Chinese, French, Spanish, Portuguese, and Korean.
- 73% newly created by professionals from scratch (modeled after tests like the Woodcock-Johnson IV and Raven's Matrices) to prevent pre-training contamination/data leakage.
- Sourced cultural content natively rather than translating from English to prevent English-centric cultural bias.
- Human Baseline Data: Collected from 480 human participants (80 native speakers per language), providing a reliable baseline for normal human cognitive performance.
Key Insights and Ideas
- Crystallized vs. Fluid Intelligence Gap: MLLMs excel at crystallized knowledge ($G_c$) and language-heavy tasks ($G_{rw}$) due to massive text pre-training, but lag severely in fluid reasoning ($G_f$) and visual-spatial processing ($G_v$).
- Evidence of a Machine "$g$-Factor": The authors observed a clear "Winner Takes All" pattern: models that excelled in one cognitive domain consistently dominated across others. This mirrors the positive correlation of cognitive abilities seen in humans (Charles Spearman’s classic general factor, $g$).
- Alignment with Real-World Human Preference: M3GIA scores achieved an $R^2 = 0.74$ linear correlation with Chatbot Arena (Vision) human Elo ratings—drastically outperforming traditional benchmarks like MMMU ($R^2 = 0.53$), MM-Vet ($R^2 = 0.55$), and MMBench ($R^2 = 0.03$).
- Vision Pre-Training Over Parameter Scale: For visual processing ($G_v$), the number of training samples seen by the visual encoder matters far more than its raw parameter size (e.g., a 303M ViT trained on 32B samples achieved 38.3% accuracy, outperforming a 632M ViT trained on the same data volume at 37.5%).
Key Results
- Overall Accuracy vs. Human: Humans averaged 76.9% overall accuracy across tasks. Among 24 evaluated models:
- Gemini-1.5-Pro scored highest at 62.4%.
- GPT-4o reached 59.8%.
- Open-source frontrunners: InternVL-chat-v1.2-plus scored 51.9%; Mini-Gemini-34B scored 48.2%.
- Visual-Spatial Processing ($G_v$): Humans scored 81.1%, while the best model (Gemini-1.5-Pro) only reached 53.8%.
- Multilingual Disparity: Normalized English GIA scores for GPT-4o hit 86.5% of the human mean, but dropped noticeably in Chinese (68.7%), French (63.3%), and Korean (72.1%).
- Emergence Threshold: Ablations on the Qwen-1.5 suite (1.8B to 72B) showed cognitive ability gains plateauing between 7B and 14B, with a sharp inflection in general cognitive ability emerging between 13B and 34B.
Limitations and Future Work
- Omission of Auditory Processing ($G_a$): Auditory processing was excluded because most current MLLMs do not natively handle audio, though the authors plan to incorporate it as multimodal audio-vision-language models mature.
- Language Coverage: Due to the high cost and labor required to handcraft expert psychometric items natively, the current benchmark is limited to six high-resource languages. Extending M3GIA to lower-resource and culturally distinct languages remains future work.
Summary updated