How Well Do Machines Perform on IQ tests: a Comparison Study on a Large-Scale Dataset

Yusen Liu, Fangyuan He, Haodi Zhang, Guozheng Rao, Zhiyong Feng, Yi Zhou

Abstract

Measuring general machine intelligence remains a significant challenge in artificial intelligence research. While specialized models excel in narrow domains, human-centric benchmarks like Intelligence Quotient (IQ) tests offer a multi-modal assessment of reasoning capabilities. This paper introduces IQ10k, the first large-scale dataset for machine IQ testing, comprising 10,007 questions across verbal, sequence, diagrammatic, and logical categories. We evaluate various state-of-the-art models against human baselines, revealing that while AI approaches human-level proficiency in verbal associations, it struggles significantly with the logical reasoning required for numerical sequences.

Introduction

While AI has achieved superhuman performance in specific tasks like image recognition and strategic games, the quest for General Artificial Intelligence (GAI) requires benchmarks that measure broader cognitive flexibility. IQ tests are the gold standard for human psychometrics, requiring a combination of natural language processing (NLP), logic, knowledge representation, and visual reasoning.

Historically, IQ datasets for AI have suffered from two major limitations: scale and diversity. Most existing datasets contain only a few hundred questions and focus on single niches, such as number sequences. To address this, we present IQ10k, a statistically significant dataset designed to measure how modern AI stacks up against the human mind.

The IQ10k Dataset

The IQ10k dataset consists of 10,007 manually curated genuine IQ questions collected from diverse sources, including academic books and specialized websites. It is designed to be roughly 40 times larger than any previous IQ dataset used in AI research, providing a robust platform for statistical analysis.

Table 1: Distribution of the 10,007 questions in the IQ10k dataset across major categories.

The dataset is divided into four major categories to test different cognitive facets:

  • Verbal (4,503 questions): Focuses on analogies, classifications, synonyms, and antonyms.
  • Sequence (2,562 questions): Tests numerical pattern recognition.
  • Diagram (1,205 questions): Involves visual puzzles like Raven’s Progressive Matrices.
  • Other (1,737 questions): Covers math, logic, and common-sense riddles.

Methodology

We evaluated several state-of-the-art (SOTA) approaches against a human baseline of 25 volunteers. For Verbal Reasoning, we tested word embedding models including Word2Vec, GloVe, ConceptNet Numberbatch, and Retrofitting. These models utilize vector mathematics to solve analogies and classification tasks.

For Sequence Reasoning, we compared four distinct methodologies:

  1. Search-based: The Online Encyclopedia of Integer Sequences (OEIS).
  2. Symbolic/Mathematical: Wolfram Mathematica.
  3. Semi-analytical: BatchSeq.
  4. Neural: A custom Artificial Neural Network (ANN) approach.

Results and Analysis

Our results highlight a significant "intelligence gap" between humans and current AI models, particularly in logical reasoning.

The Sequence Slump

Sequence reasoning proved to be the most difficult category for machines. While humans averaged 55.09% accuracy, the best machine approach (ANN) managed only 29.80%. Interestingly, high-powered mathematical engines like Mathematica and search tools like OEIS lagged further behind, scoring between 17% and 19%.

Figure 1: Performance comparison between humans and various AI approaches across different sequence reasoning sub-tasks.

This failure occurs because AI sequence solvers often search for complex mathematical polynomials, whereas IQ tests rely on "simple" human logic—such as Fibonacci sequences or digit-based patterns—that these engines are not prioritized to find.

Verbal Parity

In contrast, the verbal category showed surprising parity. In verbal classification, humans scored ~68.57%, while ConceptNet followed closely at 62.61%. In verbal analogies, ConceptNet reached 47.34% against a human baseline of ~62%. This suggests that the relational data in modern knowledge graphs is beginning to effectively mimic human-like word association.

Conclusion

While AI is beginning to close the gap in verbal reasoning, it remains far from human-level performance in logical and sequence-based tasks. The IQ10k dataset highlights the need for "Neuro-symbolic AI"—architectures that can combine sub-symbolic neural vision with symbolic logical reasoning. To encourage further research, we have established an open automated IQ test platform for the community to benchmark new algorithms against this dataset.

Summary updated

Abstract

AI benchmarking becomes an increasingly important task. As suggested by many researchers, Intelligence Quotient (IQ) tests, which is widely regarded as one of the predominant benchmarks for measuring human intelligence, raises an interesting challenge for AI systems. For better solving IQ tests automatedly by machines, one needs to use, combine and advance many areas in AI including knowledge representation and reasoning, machine learning, natural language processing and image understanding. Also, automated IQ tests provides an ideal testbed for integrating symbolic and sub-symbolic approaches as both are found useful here. Hence, we argue that IQ tests, although not suitable for testing machine intelligence, provides an excellent benchmark for the current development of AI research. Nevertheless, most existing IQ test datasets are not comprehensive enough for this purpose. As a result, the conclusions obtained are not representative. To address this issue, we create IQ10k, a large-scale dataset that contains more than 10,000 IQ test questions. We also conduct a comparison study on IQ10k with a number of state-of-the-art approaches.
BibTeX
@inproceedings{liu2019how,
  author = {Yusen Liu and Fangyuan He and Haodi Zhang and Guozheng Rao and Zhiyong Feng and Yi Zhou},
  title = {{How Well Do Machines Perform on IQ tests: a Comparison Study on a Large-Scale Dataset}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.24963/ijcai.2019/846},
}
Figure