Abstract
Measuring general machine intelligence remains a significant challenge in artificial intelligence research. While specialized models excel in narrow domains, human-centric benchmarks like Intelligence Quotient (IQ) tests offer a multi-modal assessment of reasoning capabilities. This paper introduces IQ10k, the first large-scale dataset for machine IQ testing, comprising 10,007 questions across verbal, sequence, diagrammatic, and logical categories. We evaluate various state-of-the-art models against human baselines, revealing that while AI approaches human-level proficiency in verbal associations, it struggles significantly with the logical reasoning required for numerical sequences.
Introduction
While AI has achieved superhuman performance in specific tasks like image recognition and strategic games, the quest for General Artificial Intelligence (GAI) requires benchmarks that measure broader cognitive flexibility. IQ tests are the gold standard for human psychometrics, requiring a combination of natural language processing (NLP), logic, knowledge representation, and visual reasoning.
Historically, IQ datasets for AI have suffered from two major limitations: scale and diversity. Most existing datasets contain only a few hundred questions and focus on single niches, such as number sequences. To address this, we present IQ10k, a statistically significant dataset designed to measure how modern AI stacks up against the human mind.
The IQ10k Dataset
The IQ10k dataset consists of 10,007 manually curated genuine IQ questions collected from diverse sources, including academic books and specialized websites. It is designed to be roughly 40 times larger than any previous IQ dataset used in AI research, providing a robust platform for statistical analysis.
The dataset is divided into four major categories to test different cognitive facets:
- Verbal (4,503 questions): Focuses on analogies, classifications, synonyms, and antonyms.
- Sequence (2,562 questions): Tests numerical pattern recognition.
- Diagram (1,205 questions): Involves visual puzzles like Raven’s Progressive Matrices.
- Other (1,737 questions): Covers math, logic, and common-sense riddles.
Methodology
We evaluated several state-of-the-art (SOTA) approaches against a human baseline of 25 volunteers. For Verbal Reasoning, we tested word embedding models including Word2Vec, GloVe, ConceptNet Numberbatch, and Retrofitting. These models utilize vector mathematics to solve analogies and classification tasks.
For Sequence Reasoning, we compared four distinct methodologies:
- Search-based: The Online Encyclopedia of Integer Sequences (OEIS).
- Symbolic/Mathematical: Wolfram Mathematica.
- Semi-analytical: BatchSeq.
- Neural: A custom Artificial Neural Network (ANN) approach.
Results and Analysis
Our results highlight a significant "intelligence gap" between humans and current AI models, particularly in logical reasoning.
The Sequence Slump
Sequence reasoning proved to be the most difficult category for machines. While humans averaged 55.09% accuracy, the best machine approach (ANN) managed only 29.80%. Interestingly, high-powered mathematical engines like Mathematica and search tools like OEIS lagged further behind, scoring between 17% and 19%.
This failure occurs because AI sequence solvers often search for complex mathematical polynomials, whereas IQ tests rely on "simple" human logic—such as Fibonacci sequences or digit-based patterns—that these engines are not prioritized to find.
Verbal Parity
In contrast, the verbal category showed surprising parity. In verbal classification, humans scored ~68.57%, while ConceptNet followed closely at 62.61%. In verbal analogies, ConceptNet reached 47.34% against a human baseline of ~62%. This suggests that the relational data in modern knowledge graphs is beginning to effectively mimic human-like word association.
Conclusion
While AI is beginning to close the gap in verbal reasoning, it remains far from human-level performance in logical and sequence-based tasks. The IQ10k dataset highlights the need for "Neuro-symbolic AI"—architectures that can combine sub-symbolic neural vision with symbolic logical reasoning. To encourage further research, we have established an open automated IQ test platform for the community to benchmark new algorithms against this dataset.
Summary updated