Transforming Integer Grids to Spatial Intuition: Perception, Not Planning, Governs ARC-AGI-3

Harsidak Singh BanwaitarXiv 2026 · arXiv

TL;DR

SARBLOH is a new autonomous agent architecture that achieves state-of-the-art performance on the interactive ARC-AGI-3 benchmark by treating spatial perception as the primary bottleneck for AI reasoning. By replacing raw integer grid inputs with a deterministic perception engine that extracts geometric invariants and testable hypotheses, SARBLOH achieved a 16.53 score on the official hidden Kaggle benchmark—a 13.7x improvement over the previous milestone winner.

What Problem Does This Paper Address?

The ARC-AGI-3 benchmark challenges AI agents to solve unfamiliar, uninstructed visual games. Unlike the original ARC, this version is interactive: agents must explore the environment to deduce rules. A critical hurdle is the Relative Human Action Efficiency (RHAE) metric, which penalizes unnecessary exploratory actions quadratically.

The author identifies a fundamental failure mode called Perceptual Misinduction. Most current models receive the environment as a raw matrix of integers (e.g., a 64x64 grid). Serializing this grid into text tokens destroys the "spatial topology"—the model no longer "sees" that two cells are adjacent. Consequently, frontier models squander their action budgets just trying to rediscover basic boundaries and symmetries that a human would perceive instantly.

Controlled ablation showing that providing a rendered image or board explanation drastically reduces the actions needed to solve a level compared to raw integer grids.

Methods

The core of the paper is SARBLOH, an open-weight agent (based on Qwen-3.8-Flash-Next) that shifts the burden of spatial reasoning from the neural network to a deterministic, host-side perception engine.

The Measured–Guessed Rule

SARBLOH operates on a strict dichotomy:

  1. Measured Facts: Geometric invariants like reachability, enclosures, and symmetries ($D_4 \times \mathbb{Z}^+$) are computed deterministically in $O(HW)$ time. These are presented to the model as absolute truths.
  2. Guessed Roles: Functional affordances (e.g., "Is this object the player?") are treated as probabilistic hypotheses. Each guess is paired with a specific one-action verification test to confirm or refute it immediately.

The Perception Engine

Instead of raw grids, the agent receives a "Fused Briefing" consisting of:

  • Object Inventory: A tracked list of monochromatic regions that maintains identity across frames.
  • Annotated Canvas: A visual blueprint that highlights structures like floors, walls, and HUD elements.
  • Geometric Briefing: A text summary of spatial relations (e.g., "obj 6 is reachable in 6 steps").

Code-as-Action Runtime

SARBLOH interacts via an embedded IPython kernel. Instead of emitting low-level motor primitives (like "Move Up"), it synthesizes algorithmic hypotheses in Python. It uses a speculative await play() primitive that executes multi-step trajectories but halts immediately if the environment's state deviates from the agent's internal model.

An end-to-end execution turn showing the briefing, the agent's Python code, and the resulting state change.

Datasets

  • ARC-AGI-3 Public Benchmark: 25 interactive environments used for comparative analysis against frontier models like GPT-6.1 and Claude Opus 5.
  • Official Hidden Kaggle Evaluation Suite: A set of undisclosed games used to verify the final competition score under strict 9-hour and single-GPU constraints.
  • Held-out Games: 23 official games used to test the cross-distribution generalization of the perception engine.

Key Insights and Ideas

  • Perception > Planning: Ablation studies (Figure 2) reveal that over 90% of efficiency gains come from structuring the perceptual representation rather than increasing the depth of the model's planning.
  • Quadratic Penalties Favor Internal "Thinking": Because the benchmark penalizes physical actions quadratically but does not penalize internal computation time, it is always optimal to perform extensive host-side geometric parsing and speculative planning before committing to a move.
  • Token Efficiency: By abstracting the grid into objects and relations, SARBLOH reduces the prompt token footprint by 89.4% (from ~3,500 to ~370 tokens per frame), which significantly increases the capacity of the GPU's prefix cache.

Key Results

  • Benchmark Dominance: SARBLOH achieved an official hidden Kaggle score of 16.53, advancing 13.7x beyond the first Milestone winner.
  • Surpassing Frontier Models: On public environments, SARBLOH (an open-weight model) achieved a peak mean efficiency of 72.5%, outperforming proprietary API-based models like GPT-6.1 (54.8%) and Claude Opus 5 (37.8%) on 21 of 25 games.
  • Human-Level Efficiency: In "Run B" (full briefing active), the agent recorded a median action ratio of 0.68x relative to human baselines, meaning it often solved levels with fewer actions than a median human.
Comparison of SARBLOH against five frontier API baselines across 25 public environments.

Limitations and Future Work

  • Stateless Hypotheses: The current perception engine evaluates hypotheses independently at each frame. The authors suggest moving toward a Closed-Loop Belief Updating system that maintains a Bayesian belief distribution across the entire episode.
  • General-Purpose Weights: The agent currently relies on general-purpose model weights. Offline Trajectory Fine-Tuning (LoRA) on successful, high-efficiency execution traces could further optimize code generation.
  • Reinforcement Learning: Exploring sample-efficient RL algorithms like GRPO to optimize planning depth within the fixed computational envelope of the competition.

Summary updated

Abstract

Sample-efficient fluid intelligence requires agents that rapidly form working models of unfamiliar interactive environments. ARC-AGI-3 tests this across uninstructed visual games scored by Relative Human Action Efficiency, which penalizes exploratory waste quadratically. When frontier models read raw integer grids, serialization destroys spatial topology, and they spend hundreds of actions rediscovering basic boundaries and symmetries. We introduce SARBLOH, an open-weight agent built on the premise that spatial perception, not planning capacity, governs interactive reasoning on ARC-AGI-3. A zero-parameter O(HW) perception engine turns each frame, in 35 ms, into an object inventory with bipartite temporal tracking, a geometric briefing that separates proven invariants from affordance hypotheses, and an annotated canvas. Paired with a code-as-action runtime, the agent writes algorithmic hypotheses instead of low-level moves. Across eight presentations, grounded perception raises the rubric score from 3/28 to 25/28 points while cutting token use by 89.4%. On 23 held-out games, geometric invariants generalize with 100% accuracy, while single-frame affordance guesses fall to 6-50%, so active verification probes remain necessary. On the public set, Run A (briefing off) solved 143 of 183 levels in 8.2 hours at 59.4% single-pass efficiency; Run B (briefing on) reached 63.5%. Within the nine-hour budget on one Kaggle RTX PRO 6000 GPU, SARBLOH scored 16.53 on the hidden Kaggle evaluation, 13.7x the first Milestone winner.
BibTeX
@misc{banwait2026transforming,
  author = {Harsidak Singh Banwait},
  title = {{Transforming Integer Grids to Spatial Intuition: Perception, Not Planning, Governs ARC-AGI-3}},
  year = {2026},
  eprint = {2611.06793},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.06793},
}
Figure