Introduction
Abstract reasoning, particularly Diagrammatic Abstract Reasoning (DAR), represents a pinnacle of human fluid intelligence. Unlike standard video prediction where frames change via smooth optical flow, DAR involves discrete, rule-governed transformations—such as rotations, texture changes, or numerical increments. Traditional deep learning models often struggle with these "jumps," frequently producing blurry or averaged results because they cannot decide between multiple logical possibilities. This paper introduces the Contextual RNN-GAN (Context-RNN-GAN), a framework designed to navigate these complex logical sequences and predict the next step in a visual puzzle.
Methods: The Contextual RNN-GAN Architecture
The proposed architecture moves beyond static image generation by incorporating temporal context into both the generation and evaluation phases. The core innovation is the use of Recurrent Neural Networks (RNNs) within a Generative Adversarial Network (GAN) framework.
As illustrated in the architecture diagram, the system consists of two primary components:
- The Generator (G): A Gated Recurrent Unit (GRU) based RNN that processes a sequence of diagrams ($x_1, ..., x_t$) to predict the subsequent frame ($y_t$).
- The Contextual Discriminator (D): Unlike standard GAN discriminators that evaluate images in isolation, this discriminator is itself an RNN. It analyzes the entire sequence history alongside the generated candidate to determine if the new image is a logically consistent continuation of the pattern. This "contextual" check forces the generator to adhere to the underlying rules of the sequence rather than just producing a realistic-looking shape.
To represent the images effectively, the authors introduced a Siamese CNN. This network learns a joint embedding of temporally adjacent images, capturing how components (like lines or shaded regions) evolve from one step to the next. This approach proved more effective than using static features from pre-trained models like AlexNet.
Key Insights: Overcoming the Blurriness Trap
A major challenge in DAR is the "multi-modal" nature of the predictions. Given a sequence, several logical rules might seem plausible to a naive model. Standard regression losses like L1 or L2 tend to produce a pixel-wise average of these possibilities, resulting in "ghosting" or blurry artifacts where multiple potential shapes are superimposed.
By employing an adversarial loss, the Context-RNN-GAN forces the model to commit to a single, sharp "guess" that satisfies the contextual discriminator. As shown in the comparison above, while L2 loss results in cluttered, ambiguous images, the Context-RNN-GAN (Context-R-G) produces clear, distinct diagrams that follow the established sequence logic.
Experimental Results
The model was evaluated on two primary benchmarks:
- DAT-DAR Dataset: A custom dataset of 1,500 IQ-style problems, expanded to 12,000 sequences through data augmentation (rotations and mirrors). The Context-RNN-GAN achieved an accuracy of 35.4%, nearly matching the average performance of 10th-grade students (36.67%).
- Moving MNIST: On this standard benchmark where digits bounce within a frame, the model outperformed existing state-of-the-art methods like AE-Conv-LSTM, achieving lower Cross-Entropy (241.8) and Squared Error (167.9) scores.
Conclusion
The Context-RNN-GAN demonstrates that contextual adversarial training is a powerful tool for solving abstract visual puzzles. While the model currently performs at a high-school level, it still lags behind college-level reasoning (44.17%). Future work may involve incorporating explicit "object-level" reasoning to handle even more complex logical transformations and discontinuous movements.
Summary updated