Abstract
Standard Vision Transformers (ViTs) fail to solve the Abstraction and Reasoning Corpus (ARC) puzzles, even when trained on millions of examples. This paper introduces ViTARC, a modified architecture that achieves a 75% solve rate by imbuing the model with critical 2D inductive biases, including spatially-aware "border tokens," relative positional encodings, and object-based priors.
Introduction: The Challenge of Fluid Intelligence
The Abstraction and Reasoning Corpus (ARC) consists of 2D grid puzzles designed to measure human-like fluid intelligence. Unlike standard computer vision tasks, ARC requires reasoning about novel rules from just a few examples. While most research attempts to solve ARC via program synthesis, this study investigates whether a Vision Transformer (ViT) can learn these mappings directly through supervised learning.
Surprisingly, a "vanilla" ViT—the state-of-the-art for natural images—fails on 90% of tasks even when trained on one million examples per task. This failure highlights a fundamental lack of spatial awareness and object-centric reasoning in standard Transformer architectures.
ViTARC: Bridging the Representational Gap
To address these shortcomings, the authors propose ViTARC. This model introduces several architectural modifications designed to preserve 2D structure and facilitate object-level reasoning.
Key modifications include:
- Pixel-Level Tokenization: Every 1x1 pixel is treated as an independent token to maintain maximum precision.
- Spatially-Aware "Border Tokens": The introduction of
<2d_nl>(newline) and<2d_endxgrid>tokens helps the model reconstruct the 2D grid boundaries after flattening the image into a 1D sequence. - Positional Encoding Mixer (PEmixer): Learnable weights balance the importance of color (pixel value) versus location (positional embedding).
- 2D Relative Positional Encoding (2D-RPE): Explicit calculation of Manhattan distances between pixels captures local geometric relationships.
- Object-based Positional Encoding (OPE): Automated segmentation assigns "object IDs" to pixel groups, allowing the model to reason about discrete entities.
Experimental Setup and Datasets
The study utilizes 400 public tasks from the ARC benchmark. To overcome the data scarcity of the original few-shot setting, the researchers used the RE-ARC generator to procedurally create one million unique input-output pairs for each task. This allows for a rigorous evaluation of the architectural limits of Transformers in a supervised setting.
Results and Performance Analysis
The results demonstrate that architectural inductive biases are far more important than raw data volume for ARC tasks.
The progression of performance across the 400 tasks is as follows:
- Vanilla ViT: 17.68% mean solve rate.
- ViTARC-VT (with 2D tokens/padding): 66.03% mean solve rate.
- Full ViTARC (with PEmixer, 2D-RPE, and OPE): 75.04% mean solve rate.
Notably, the full ViTARC model achieved a test solve rate close to 100% on more than half of the tasks. This suggests that the "1D flattening trap" of standard ViTs—where exact relative positions are lost—is a primary bottleneck in visual reasoning.
Conclusion
The success of ViTARC proves that Transformers can master complex visual logic if provided with the correct spatial "vocabulary." The fact that a model with one million training examples still fails without these 2D modifications suggests that "more data" is not a substitute for sound architectural priors. Future work will focus on cross-task generalization, using ViTARC as a foundation for multi-task vision models capable of solving unseen ARC puzzles.
Summary updated