Abstract Diagrammatic Reasoning with Multiplex Graph Networks

Duo Wang, Mateja Jamnik, Pietro Liò

Introduction

Researchers from the University of Cambridge have introduced MXGNet, a novel graph neural network that masters abstract visual reasoning tasks like Raven’s Progressive Matrices. By representing diagrams as "multiplex graphs"—where multiple types of relationships exist between objects across different image panels—the model significantly outperforms previous state-of-the-art systems in both accuracy and the ability to generalize to unseen logic.

The Challenge of Abstract Reasoning

Abstract reasoning is a cornerstone of human intelligence. When we look at a puzzle, we don't just see pixels; we identify objects (like circles and squares) and infer the hidden rules governing them (e.g., "the number of shapes increases by one in every row").

For Artificial Intelligence, this is notoriously difficult. Standard Convolutional Neural Networks (CNNs) excel at recognizing what is in an image, but they struggle to understand the relationships between multiple images. Previous models like the Wild Relation Network (WReN) made progress, but a significant performance gap remained. The core challenge is twofold:

  1. Object Identification: Isolating individual elements within a complex scene.
  2. Relational Logic: Identifying how those elements change across a sequence of panels (rows or columns) according to logical rules like XOR, OR, or arithmetic progressions.

Methods: Introducing MXGNet

To tackle these challenges, the authors propose MXGNet (Multiplex Graph Network), which combines several sophisticated concepts to process and reason over visual data.

The MXGNet architecture, illustrating the transformation of context panels into multilayer graphs for reasoning.
  • Object-Level Representation: Instead of processing a whole image as one block, MXGNet extracts features for every individual shape. The researchers experimented with two methods: a CNN grid (treating spatial locations as objects) and a Spatial Attention mechanism (which explicitly "looks" for specific objects).
  • Multiplex Graphs: In a standard graph, an edge between two nodes represents a single connection. In a multiplex graph, an edge can represent multiple "layers" of relationships simultaneously—such as color, shape, and position. MXGNet treats each diagram panel as a layer and builds connections across them to capture how attributes evolve.
Node aggregation across different diagrams, demonstrating how multiplex edges capture relationships across panels.
  • Search Space Reduction: To avoid checking every possible combination of panels (which is computationally expensive), the authors implemented a "gating" mechanism. This pre-training step allows the model to automatically discover that it should focus on rows and columns to find the logic, effectively "hard-gating" out irrelevant noise.
  • Reasoning Network: Finally, the summarized graph data is fed into a residual neural network that compares the context panels against candidate answers to pick the most logical fit.

Datasets and Evaluation

The model was rigorously evaluated on three major benchmarks:

  1. PGM (Procedurally Generated Matrices): A massive dataset of 1.2 million tasks involving complex logical rules.
  2. RAVEN: A similar RPM-style dataset but with more diverse and hierarchical image structures.
  3. Diagram Syllogisms: A logic task using Euler diagrams (circles representing sets) to infer conclusions from two premises.

Key Insights

  • Relational Inductive Bias: By utilizing a graph structure, the authors bake the concept of "relationships" directly into the architecture. This is far more efficient than asking a standard neural net to "figure out" that objects relate to one another from scratch.
  • The Power of Multiplexity: Real-world reasoning isn't one-dimensional. An object might change its color and its position at the same time. The multiplex approach allows the model to track these parallel changes without them getting tangled in a single feature vector.
  • Generalization is Key: The paper places a heavy emphasis on "extrapolation"—testing the model on logic it hasn't seen before (e.g., training on small shapes and testing on large ones). MXGNet showed a much smaller performance drop when moving from training to testing compared to its predecessors.

Key Results

MXGNet set new benchmarks across the board:

  • PGM Dataset: Achieved 89.6% accuracy on the neutral split, a massive jump over the previous state-of-the-art (WReN) at 76.9%.
  • RAVEN Dataset: Reached 83.91% accuracy, nearly doubling the performance of some previous models.
  • Euler Syllogisms: Achieved a near-perfect 99.8% accuracy.
  • Generalization: In "interpolation" tests (filling in missing values within a known range), MXGNet outperformed WReN by a significant margin (65.4% vs 64.4% at $\beta=0$, and even higher with auxiliary training).

Limitations and Future Work

Despite its success, the authors note a few hurdles:

  • Interpretability: While the model is highly accurate, the internal representations are still "entangled." We know it works, but we cannot yet easily "read out" the exact logical rule the model has discovered in plain English.
  • Memory vs. Reasoning: The model struggled with the "ConsistentUnion" rule in PGM. The authors suggest this specific rule acts more like a memory task than a relational reasoning task, indicating that MXGNet is specialized for logic rather than rote memorization.
  • Real-World Scaling: The next step is moving beyond synthetic diagrams to real-world applications, such as a robot identifying how LEGO bricks fit together by observing a series of assembly steps.

Summary updated

Abstract

reasoning, particularly in the visual domain, is a complex human ability, but it remains a challenging problem for artificial neural learning systems. In this work we propose MXGNet, a multilayer graph neural network for multi-panel diagrammatic reasoning tasks. MXGNet combines three powerful concepts, namely, object-level representation, graph neural networks and multiplex graphs, for solving visual reasoning tasks. MXGNet first extracts object-level representations for each element in all panels of the diagrams, and then forms a multi-layer multiplex graph capturing multiple relations between objects across different diagram panels. MXGNet summarises the multiple graphs extracted from the diagrams of the task, and uses this summarisation to pick the most probable answer from the given candidates. We have tested MXGNet on two types of diagrammatic reasoning tasks, namely Diagram Syllogisms and Raven Progressive Matrices (RPM). For an Euler Diagram Syllogism task MXGNet achieves state-of-the-art accuracy of 99.8%. For PGM and RAVEN, two comprehensive datasets for RPM reasoning, MXGNet outperforms the state-of-the-art models by a considerable margin.
BibTeX
@inproceedings{wang2020abstract,
  author = {Duo Wang and Mateja Jamnik and Pietro Liò},
  title = {{Abstract Diagrammatic Reasoning with Multiplex Graph Networks}},
  booktitle = {International Conference on Learning Representations},
  year = {2020},
}
Figure