Solving Visual Analogies Using Neural Algorithmic Reasoning (Student Abstract)

Atharv Sonwane, Gautam Shroff, Lovekesh Vig, Ashwin Srinivasan, Tirtharaj DashAAAI 2022 · OpenReview

Introduction

Visual analogical reasoning—the ability to identify that "Image A is to Image B as Image C is to Image D"—is a cornerstone of human intelligence. For a computer, this requires discovering an underlying transformation procedure from just a few examples. While symbolic search (finding a sequence of predefined code-like steps) can solve these problems, it often lacks the flexibility to handle raw visual data. Conversely, standard deep learning models often struggle with the strict, step-by-step logic required for complex transformations. This paper addresses this gap by bridging symbolic program synthesis with neural algorithmic reasoning, using neural networks to represent the individual steps in a program.

Methodology

The authors employ a variation of the Neural Algorithmic Reasoning framework, which decouples how an object is represented from how it is transformed. The pipeline consists of three main stages:

1. Representation Learning

The system first trains an autoencoder on symbolic descriptions (e.g., multi-hot vectors representing shape and position) to create a well-structured latent space. A CNN-based encoder is then trained to map raw images into this same latent space. To handle novelty, an "unseen" label is included during training, allowing the encoder to categorize unfamiliar shapes into a specific part of the latent space.

2. Transform Training and Program Synthesis

Instead of symbolic code, single-hidden-layer MLPs are used to represent elementary transformations like shift-right or to-square. These networks are trained to manipulate the latent vectors directly. A "program" is defined as a sequence of these neural primitives (plus operations like reset and out). Given an input/output pair, the system performs an exhaustive search to find the sequence of neural transforms that correctly maps the input to the output.

Overview of the neural-symbolic search process for solving visual analogies.

Experimental Evaluation

The researchers used a custom-generated dataset of simple shapes on a $3 \times 3$ grid to evaluate the approach. The dataset included over 16,000 programs involving positional shifts and shape conversions.

Key Results and Generalization

The system successfully found valid programs for the entire test set and maintained performance for programs up to length five, proving that neural transforms do not "drift" when applied sequentially. A major finding was the system's robust generalization: even when only a few shapes were seen during training, the system could generalize learned transformations to entirely new shapes.

Generalization performance (ratio) for encoder and transform components as more shapes are seen during training.

Discussion and Conclusion

The success of this hybrid approach relies on the power of a structured latent space. By training the latent space on symbolic descriptions first, the neural transforms can maintain "conceptual integrity" even when chained together. This decoupling of feature extraction from manipulation allows the system to learn rules that apply to any shape, including those never seen before.

While effective, the system currently relies on classical pre-processing for shape isolation. Future work aims to move toward a fully end-to-end neural approach and scale the system to more complex, real-world imagery and diverse transformations. The ultimate goal is "out-of-domain" generalization, where reasoning rules learned in one context can be applied to completely different visual domains.

Summary updated

BibTeX
@inproceedings{sonwane2022solving,
  author = {Atharv Sonwane and Gautam Shroff and Lovekesh Vig and Ashwin Srinivasan and Tirtharaj Dash},
  title = {{Solving Visual Analogies Using Neural Algorithmic Reasoning (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  url = {https://openreview.net/forum?id=mPfJ_WHFPW5},
}

Loading related content...

Figure