Introduction
For over half a century, the quest to replicate human intelligence has often used psychometric tests—standardized IQ assessments—as a primary yardstick. This paper, originally presented at IJCAI-17, provides a comprehensive meta-analysis of approximately 30 computational models developed between 1965 and 2016. It explores the evolution of these systems, the techniques employed, and the philosophical implications of machines "passing" tests designed for humans.
The Evolution of AI Psychometrics
The field has experienced a significant surge in activity recently; the authors note that half of all relevant research was published between 2011 and 2016. This "boom" reflects a broader trend in AI where benchmarks like Raven’s Progressive Matrices (RPM) and series completion are increasingly used to validate progress. However, the authors identify a critical tension: does solving an IQ test signify genuine intelligence, or merely the successful execution of a specialized algorithm? As AI hits milestones in games like Go or Jeopardy!, the community must determine if the underlying mechanisms of these solvers relate to human cognitive processes or if they are merely "gaming" the evaluation metrics.
Methodological Framework and Benchmarks
The survey evaluates models based on their range of tasks, degree of specialization, and the AI techniques utilized—ranging from early symbolic reasoning and cognitive architectures (like ACT-R) to modern deep learning and large-scale neural networks.
The "datasets" in this context are the psychometric tests themselves. The paper highlights several recurring benchmarks:
- Raven’s Progressive Matrices (RPM): A non-verbal test of abstract reasoning using 3x3 grids of patterns.
- Geometric Analogies: The classic "A is to B as C is to D" visual problems.
- Series Completion: Number or letter sequences (e.g., 1, 2, 4, 8, ...).
- Odd-One-Out: Identifying the outlier in a set of objects or patterns.
- Verbal/Lexical Analogies: Word-based reasoning (e.g., proportional analogies).
- Bongard Problems: Complex visual pattern recognition tasks.
Key Insights: The Specialization Trap
A central finding is the "Specialization Trap." Most models are "one-trick ponies" designed for a specific test type. While these systems may achieve high scores, their success does not necessarily imply general intelligence. The authors warn against the "Big Switch" approach—a meta-program that routes different problems to specialized sub-solvers. This creates an illusion of generality without the underlying cognitive flexibility.
Some sophisticated models, such as gErl or the 2.5-million-neuron Spaun model, attempt to go further by modeling "item difficulty." These systems don't just seek the correct answer; they aim to replicate human error patterns and cognitive constraints, such as working memory limits or the specific number of rules involved in a problem.
Limitations and the Path Forward
The current landscape faces several hurdles that prevent a true measurement of machine intelligence:
- Evaluation Gap: Many researchers use static, public datasets, leading to potential "rote learning" or overfitting by developers.
- Proprietary Barriers: Many official intelligence tests are not accessible for AI training, forcing the use of non-equivalent "alternative formulations."
- Static vs. Dynamic Testing: To prevent "cheating" via hard-coding, the authors advocate for instance generators—tools that create novel, unseen problems on the fly.
Conclusion: Toward Universal Psychometrics
The paper concludes by proposing a shift toward "universal psychometrics." This emerging discipline seeks to develop tests applicable to any agent—human, animal, robot, or swarm—moving beyond anthropocentric IQ tests. By focusing on a "decathlon" of diverse, dynamically generated problems and a more theoretical framework, the AI community can move closer to measuring true general artificial intelligence rather than just specialized task performance.
Summary updated