Duel-based Deep Learning system for solving IQ tests

Paulina Tomaszewska, Adam Zychowski, Jacek MandziukAISTATS 2022 · OpenReview

Abstract

One of the relevant aspects of Artificial General Intelligence is the ability of machines to demonstrate abstract reasoning skills, for instance, through solving (human) IQ tests. This work presents a new approach to machine IQ tests solving formulated as Raven’s Progressive Matrices (RPMs), called Duel-IQ. The proposed solution incorporates the concept of a tournament in which the best answer is chosen based on a set of duels between candidate RPM answers. The three relevant aspects are: (1) low computational and design complexity, (2) proposition of two schemes of pairing up candidate answers for the duels and (3) evaluation of the system on a dataset of shapes other than those used for training. Depending on a particular variant, the system reaches up to $82.8%$ accuracy on average in RPM tasks with 5 candidate answers and is on par with human performance and superior to other literature approaches of comparable complexity when training and test sets are from the same distribution.

Introduction

Solving visual IQ tests like Raven’s Progressive Matrices (RPMs) serves as a critical benchmark for Artificial General Intelligence (AGI) because it requires abstract reasoning rather than simple pattern recognition. In a standard RPM task, a subject is presented with a 3x3 grid of images where the final panel is missing. The goal is to select the correct filler from a set of candidates based on underlying logical rules.

The core challenge in these tests is relativity. Traditional machine learning models often treat this as a binary classification problem ("Is this answer correct?") or a multi-class problem. However, these approaches often fail to capture the comparative nature of human reasoning, where possibilities are weighed against each other to determine which logical explanation is most convincing. This paper introduces Duel-IQ, a system that frames the problem as a tournament of pairwise comparisons.

Methodology

System Architecture

The Duel-IQ pipeline consists of four primary stages: preprocessing, feature extraction, the duel (classification), and scoring. Initially, the system generates "filled-in" versions of the RPM for every possible candidate answer. These high-dimensional images are then compressed into a latent representation using a convolutional autoencoder, ensuring the model focuses on structural features rather than pixel-level noise.

The Duel-IQ system architecture showing the tournament-style scoring module.

The Duel Mechanism and Auxiliary Training

The heart of the system is the classification network, which performs "duels" between candidate answers. Instead of evaluating a single answer in isolation, the model takes two filled-in matrices and determines which is more likely to be correct. The authors explored two modes: winner_in (where one answer is guaranteed to be correct) and any_in (a three-class setup allowing for draws).

To sharpen the model's ability to distinguish subtle geometric differences, the system utilizes auxiliary training. The model is simultaneously tasked with calculating a "dissimilarity measure" between candidates, which forces the network to learn fine-grained features of shape, size, and rotation.

The classification network architecture featuring dual encoders and an auxiliary output for dissimilarity measures.

Scoring and Aggregation

After running a round-robin tournament where every candidate duels every other candidate, the results are aggregated. The system uses either Probability Summing (aggregating softmax scores) or Winning Frequency (counting raw wins) to select the final system answer.

Experimental Results

Performance and Human Baseline

The system was evaluated on a custom-generated dataset involving logic rules for shape, size, and rotation. Duel-IQ achieved a peak accuracy of 82.8% on the standard test set. For comparison, a survey of 94 university students and PhD candidates yielded a human average of 77.3%. Duel-IQ not only matched but slightly exceeded human performance in this specific synthetic environment, outperforming established baselines like WReN (Wild Relation Network).

Generalization Challenges

In a second experiment designed to test generalization, the model was trained on one set of shapes (e.g., circles, squares) and tested on entirely different geometries (e.g., hearts, stars). In this out-of-distribution scenario, accuracy dropped to approximately 49%. While significantly better than random chance (20%), this gap suggests the model remains somewhat dependent on the visual features of the training set rather than achieving pure logical abstraction.

Conclusion

Duel-IQ demonstrates that a tournament-based comparative approach is a powerful paradigm for abstract reasoning tasks. By focusing on the relative fitness of answers, the system achieves state-of-the-art results on standard RPM benchmarks. Future work will focus on improving out-of-distribution generalization by incorporating more robust, shape-invariant feature extractors and testing on more complex, real-world hierarchical datasets.

Summary updated

Abstract

One of the relevant aspects of Artificial General Intelligence is the ability of machines to demonstrate abstract reasoning skills, for instance, through solving (human) IQ tests. This work presents a new approach to machine IQ tests solving formulated as Raven’s Progressive Matrices (RPMs), called Duel-IQ. The proposed solution incorporates the concept of a tournament in which the best answer is chosen based on a set of duels between candidate RPM answers. The three relevant aspects are: (1) low computational and design complexity, (2) proposition of two schemes of pairing up candidate answers for the duels and (3) evaluation of the system on a dataset of shapes other than those used for training. Depending on a particular variant, the system reaches up to $82.8%$ accuracy on average in RPM tasks with 5 candidate answers and is on par with human performance and superior to other literature approaches of comparable complexity when training and test sets are from the same distribution.
BibTeX
@inproceedings{tomaszewska2022duel,
  author = {Paulina Tomaszewska and Adam Zychowski and Jacek Mandziuk},
  title = {{Duel-based Deep Learning system for solving IQ tests}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2022},
  url = {https://openreview.net/forum?id=IY1nEWJ9QWx},
}
Figure