Hierarchical ConViT with Attention-Based Relational Reasoner for Visual Analogical Reasoning

Wentao He, Jialu Zhang, Jianfeng Ren, Ruibin Bai, Xudong Jiang

Solving a Raven’s Progressive Matrix (RPM)—a 3x3 grid of shapes where one must identify the missing piece—is a hallmark of human fluid intelligence. For artificial intelligence, this task presents a dual challenge: "visual perception" to identify shapes and "logical reasoning" to deduce underlying rules such as progression or arithmetic.

Researchers from the University of Nottingham Ningbo China and Nanyang Technological University have introduced HCV-ARR, a model designed to bridge the gap between perception and cognition. This hybrid architecture sets new benchmarks in visual analogical reasoning by integrating the strengths of convolutional and transformer-based networks.

The Challenge of Global Reasoning

Most existing AI models for RPMs rely on shallow Convolutional Neural Networks (CNNs). While CNNs excel at identifying local features like shape or color, they often struggle with "globality"—the complex spatial layouts and long-range dependencies that define matrix rules (e.g., "the number of shapes increases by one in every row").

Conversely, pure Vision Transformers (ViTs) are adept at capturing global dependencies but often overlook fine-grained local details. Furthermore, traditional reasoning modules frequently rely on simple row similarities, which fail to capture non-linear rules like "Distribute Three" or "Arithmetic."

Methodology: Hierarchical ConViT and ARR

The proposed HCV-ARR framework introduces two primary innovations to address these limitations:

  1. Hierarchical ConViT (HCV): This dual-branch perception module combines a Convolution Branch for low-level attributes (color, type, size) and a Transformer Branch for high-level semantics (pattern formations, positions). The model operates in multiple stages; shallow layers focus on fine details, while deeper layers progressively "zoom out" to understand overall image semantics.
  2. Attention-Based Relational Reasoner (ARR): Unlike standard reasoning modules, the ARR employs an element-wise attention mechanism. It learns specific "kernels" that weigh different dimensions of the feature vector independently. This allows the model to apply distinct logic to different attributes simultaneously—for instance, recognizing that "color" remains constant while "number" follows a progression.
Detailed architecture of the HCV-ARR framework, showcasing the dual-branch ConViT stages and the element-wise attention-based reasoning module.

Experimental Results

The HCV-ARR model was evaluated on major RPM benchmarks, including RAVEN, I-RAVEN, and RAVEN-FAIR. The results demonstrate a significant performance leap over previous state-of-the-art (SOTA) methods.

On the RAVEN-FAIR dataset, the model achieved 95.4% accuracy, a substantial improvement over the previous SOTA (MRNet) at 86.8%. The model showed particular strength in the most difficult configurations, such as the 3x3Grid and 2x2Grid, where spatial reasoning is most intense.

Comparison of accuracy (%) across different configurations on the RAVEN-FAIR dataset, highlighting the superior performance of the proposed HCV-ARR model.

Key Insights

  • Hybridization is Essential: Ablation studies revealed that using ViT alone resulted in poor performance (44.9%), confirming that local convolutional perception is a necessary foundation for global reasoning.
  • Multi-scale Receptive Fields: The hierarchical approach mimics human vision by analyzing individual shapes before synthesizing the layout of the entire grid.
  • Dimension-Specific Logic: By allowing the attention mechanism to operate element-wise, the model can uncover different rules for different visual dimensions, such as shape vs. position.

Conclusion

HCV-ARR represents a significant step forward in visual analogical reasoning. By combining hierarchical perception with attribute-specific reasoning, the model effectively handles complex spatial configurations. Future work may extend this hybrid approach to real-world visual understanding tasks, such as complex scene analysis or video activity recognition, where both local actions and global contexts are vital.

Summary updated

Abstract

Raven’s Progressive Matrices (RPMs) have been widely used to evaluate the visual reasoning ability of humans. To tackle the challenges of visual perception and logic reasoning on RPMs, we propose a Hierarchical ConViT with Attention-based Relational Reasoner (HCV-ARR). Traditional solution methods often apply relatively shallow convolution networks to visually perceive shape patterns in RPM images, which may not fully model the long-range dependencies of complex pattern combinations in RPMs. The proposed ConViT consists of a convolutional block to capture the low-level attributes of visual patterns, and a transformer block to capture the high-level image semantics such as pattern formations. Furthermore, the proposed hierarchical ConViT captures visual features from multiple receptive fields, where the shallow layers focus on the image fine details while the deeper layers focus on the image semantics. To better model the underlying reasoning rules embedded in RPM images, an Attention-based Relational Reasoner (ARR) is proposed to establish the underlying relations among images. The proposed ARR well exploits the hidden relations among question images through the developed element-wise attentive reasoner. Experimental results on three RPM datasets demonstrate that the proposed HCV-ARR achieves a significant performance gain compared with the state-of-the-art models. The source code is available at: https://github.com/wentaoheunnc/HCV-ARR.
BibTeX
@inproceedings{he2023hierarchical,
  author = {Wentao He and Jialu Zhang and Jianfeng Ren and Ruibin Bai and Xudong Jiang},
  title = {{Hierarchical ConViT with Attention-Based Relational Reasoner for Visual Analogical Reasoning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i1.25072},
}
Figure