TL;DR
Vis2State is a training-free agentic framework that solves complex visual puzzles by decoupling perception from reasoning through an explicit, revisable "problem state." By representing visual evidence as a JSON object and solving it with generated Python code that has no direct image access, the system achieves state-of-the-art performance across four major visual reasoning benchmarks.
What Problem Does This Paper Address?
Solving a visual puzzle—like finding the shortest path in a maze or calculating the minimum moves in a "Move Box" game—requires two distinct skills: accurate perception (where are the walls?) and reliable computation (what is the shortest path algorithm?).
Existing methods like Chain-of-Thought (CoT) or visual programming often entangle these two. If a model misinterprets a visual detail (e.g., misreading a coordinate), that error propagates through the entire reasoning chain. Because the "facts" and the "logic" are woven together in the model's output, it is difficult for the system to inspect its own assumptions or revise a single misinterpreted constraint without starting over. The authors argue that visual reasoning needs a clear interface—an explicit state abstraction—to separate what the model sees from how it thinks.
Methods
Vis2State introduces an agentic loop centered around a revisable problem state. The framework consists of three main components:
- Visual Inspection: The model uses a suite of tools (like
crop_region,pixel_probe, andcolor_components) to gather evidence from the image. This grounds the reasoning in concrete visual data rather than vague global impressions. - State Abstraction: This is the core innovation. The model constructs a task-dependent JSON object (the "State") that encodes objects, relations, and constraints (e.g.,
{"box": [2,3], "goal": [3,4], "walls": [[1,3]]}). - Execution and Feedback: The model generates a Python program to solve the puzzle. Crucially, this program only takes the JSON state as input and has no access to the original image.
If the execution fails or produces an error, the agent can go back, re-inspect the image, and refine either the state or the program independently. This iterative process allows the model to correct "misreads" without breaking its logical algorithm.
Datasets
The authors evaluated Vis2State on 5,968 multiple-choice questions across four diverse benchmarks:
- AlgoPuzzleVQA: Algorithmic puzzles like mazes, box pushing, and the Tower of Hanoi.
- VisualPuzzles: Questions spanning algorithmic, analogical, deductive, inductive, and spatial reasoning.
- VisuLogic: Vision-centric logic questions involving attributes, positions, and styles.
- PuzzleVQA: Abstract pattern induction puzzles covering numbers, colors, shapes, and sizes.
Key Insights and Ideas
- The State as an API: By forcing the solver code to work only on the JSON state, the authors create a "firewall" that prevents perceptual errors from being hidden within complex code logic.
- Revisability is Key: The agentic loop allows for "Aha!" moments. If a program finds no path in a maze, the agent can realize, "Maybe I missed a door," and go back to inspect specific coordinates to update the state.
- Training-Free Power: Vis2State doesn't require fine-tuning. It leverages the inherent tool-use and coding capabilities of modern Large Vision-Language Models (LVLMs) like DeepSeek V4.1 Flash and GLM-5.3-Flash.
- Perception is the Bottleneck: Error analysis (shown in Figure 4) reveals that even with this framework, "Perception" remains the largest source of error, though Vis2State corrects significantly more perceptual errors than standard CoT.
Key Results
Vis2State achieved state-of-the-art accuracy on all four benchmarks, significantly outperforming "Direct CoT" (standard prompting):
- AlgoPuzzleVQA: 97.8% accuracy (+15.5 percentage points over Direct CoT).
- VisualPuzzles: 75.0% accuracy (+13.4 pp).
- VisuLogic: 49.1% accuracy (+13.6 pp).
- PuzzleVQA: 93.3% accuracy (+8.1 pp).
The framework proved robust across different models; when switching the underlying engine from DeepSeek to GLM, Vis2State consistently provided gains of 6.9 to 12.2 percentage points.
Limitations and Future Work
- Computational Cost: Vis2State is more "expensive" than standard CoT, using 1.7–2.8× more output tokens due to the iterative inspection and coding steps.
- State-Code Mismatch: While the state is explicit, the generated program might still fail to use all relevant fields or misinterpret the rules of the puzzle.
- Perceptual Ambiguity: Some visual relations remain difficult for current models to resolve even with specialized inspection tools.
- Future Directions: The authors suggest that future work could involve more sophisticated verification steps, where the model automatically checks for inconsistencies between the constructed state and the visual evidence before attempting to solve the problem.
Summary updated