Tackling the Abstraction and Reasoning Corpus with Vision Transformers: the Importance of 2D Representation, Positions, and Objects

Wenhao Li, Yudong Xu, Scott Sanner, Elias Boutros Khalil

Abstract

Standard Vision Transformers (ViTs) fail to solve the Abstraction and Reasoning Corpus (ARC) puzzles, even when trained on millions of examples. This paper introduces ViTARC, a modified architecture that achieves a 75% solve rate by imbuing the model with critical 2D inductive biases, including spatially-aware "border tokens," relative positional encodings, and object-based priors.

Introduction: The Challenge of Fluid Intelligence

The Abstraction and Reasoning Corpus (ARC) consists of 2D grid puzzles designed to measure human-like fluid intelligence. Unlike standard computer vision tasks, ARC requires reasoning about novel rules from just a few examples. While most research attempts to solve ARC via program synthesis, this study investigates whether a Vision Transformer (ViT) can learn these mappings directly through supervised learning.

Surprisingly, a "vanilla" ViT—the state-of-the-art for natural images—fails on 90% of tasks even when trained on one million examples per task. This failure highlights a fundamental lack of spatial awareness and object-centric reasoning in standard Transformer architectures.

ViTARC: Bridging the Representational Gap

To address these shortcomings, the authors propose ViTARC. This model introduces several architectural modifications designed to preserve 2D structure and facilitate object-level reasoning.

The ViTARC architecture, illustrating the transition from 2D grid input to a flattened sequence with specialized tokens and positional encodings.

Key modifications include:

  • Pixel-Level Tokenization: Every 1x1 pixel is treated as an independent token to maintain maximum precision.
  • Spatially-Aware "Border Tokens": The introduction of <2d_nl> (newline) and <2d_endxgrid> tokens helps the model reconstruct the 2D grid boundaries after flattening the image into a 1D sequence.
  • Positional Encoding Mixer (PEmixer): Learnable weights balance the importance of color (pixel value) versus location (positional embedding).
  • 2D Relative Positional Encoding (2D-RPE): Explicit calculation of Manhattan distances between pixels captures local geometric relationships.
  • Object-based Positional Encoding (OPE): Automated segmentation assigns "object IDs" to pixel groups, allowing the model to reason about discrete entities.

Experimental Setup and Datasets

The study utilizes 400 public tasks from the ARC benchmark. To overcome the data scarcity of the original few-shot setting, the researchers used the RE-ARC generator to procedurally create one million unique input-output pairs for each task. This allows for a rigorous evaluation of the architectural limits of Transformers in a supervised setting.

Results and Performance Analysis

The results demonstrate that architectural inductive biases are far more important than raw data volume for ARC tasks.

Distribution of solved test instances per ARC task, comparing the vanilla ViT against the proposed ViTARC variants.

The progression of performance across the 400 tasks is as follows:

  • Vanilla ViT: 17.68% mean solve rate.
  • ViTARC-VT (with 2D tokens/padding): 66.03% mean solve rate.
  • Full ViTARC (with PEmixer, 2D-RPE, and OPE): 75.04% mean solve rate.

Notably, the full ViTARC model achieved a test solve rate close to 100% on more than half of the tasks. This suggests that the "1D flattening trap" of standard ViTs—where exact relative positions are lost—is a primary bottleneck in visual reasoning.

Conclusion

The success of ViTARC proves that Transformers can master complex visual logic if provided with the correct spatial "vocabulary." The fact that a model with one million training examples still fails without these 2D modifications suggests that "more data" is not a substitute for sound architectural priors. Future work will focus on cross-task generalization, using ViTARC as a foundation for multi-task vision models capable of solving unseen ARC puzzles.

Summary updated

Abstract

The Abstraction and Reasoning Corpus (ARC) is a popular benchmark focused on visual reasoning in the evaluation of Artificial Intelligence systems. In its original framing, an ARC task requires solving a program synthesis problem over small 2D images using a few input-output training pairs. In this work, we adopt the recently popular data-driven approach to the ARC and ask whether a Vision Transformer (ViT) can learn the implicit mapping, from input image to output image, that underlies the task. We show that a ViT -- otherwise a state-of-the-art model for images -- fails dramatically on most ARC tasks even when trained on one million examples per task. This points to an inherent representational deficiency of the ViT architecture that makes it incapable of uncovering the simple structured mappings underlying the ARC tasks. Building on these insights, we propose ViTARC, a ViT-style architecture that unlocks some of the visual reasoning capabilities required by the ARC. Specifically, we use a pixel-level input representation, design a spatially-aware tokenization scheme, and introduce a novel object-based positional encoding that leverages automatic segmentation, among other enhancements. Our task-specific ViTARC models achieve a test solve rate close to 100% on more than half of the 400 public ARC tasks strictly through supervised learning from input-output grids. This calls attention to the importance of imbuing the powerful (Vision) Transformer with the correct inductive biases for abstract visual reasoning that are critical even when the training data is plentiful and the mapping is noise-free. Hence, ViTARC provides a strong foundation for future research in visual reasoning using transformer-based architectures.
BibTeX
@article{li2025tackling,
  author = {Wenhao Li and Yudong Xu and Scott Sanner and Elias Boutros Khalil},
  title = {{Tackling the Abstraction and Reasoning Corpus with Vision Transformers: the Importance of 2D Representation, Positions, and Objects}},
  journal = {Accepted by TMLR},
  year = {2025},
  url = {https://openreview.net/forum?id=Al72Fp0rCg},
}
Figure