Does Writing Code Crowd Out Spatial Logic? A Tiered Test on Visual Program Synthesis

When a vision-language model looks at a picture of a fractal and writes a program to redraw it, the program usually runs but rarely draws the right picture. FractalBench found that 76.1% of generated programs were executable, yet only 5.5% of those reproduced the image, for 4.2% end-to-end success. Two explanations fit that number. The model may simply not see or understand the geometry: the turn angles, the scale ratio, the branching pattern. Or the effort of producing well-formed code may crowd out the spatial reasoning, even when the code comes out valid. This second effect, which we call syntax-logic interference, has been documented for human novices (Learnable programming: blocks and beyond) and for text-only code models (Beyond Code). It has not been tested in the multimodal setting, where the logic has to come from pixels.

We propose a controlled experiment that separates the two explanations. First, we score each generated program against the known parameters of the fractal that produced the image, tier by tier: base shape, angles and ratios, recursive structure, and depth. This replaces an all-or-nothing pixel score with a diagnosis of where the program went wrong. Second, we hold the required logic fixed and vary only how much syntax the model has to produce. The levels are free Python, a minimal turtle API, grammar-constrained decoding, and a block-style tree notation in which invalid programs cannot be written. If interference is real, the logic tiers should improve as the syntax burden drops, even among programs that already run. If they do not improve, the field can stop blaming notation and focus on spatial abstraction.

Core Claim

The claim can be tested and could fail. For images where the model demonstrably perceives the geometry correctly, lowering the syntactic burden of the output notation will raise accuracy on angle, recursion-structure and depth recovery. The improvement will not be explained by fewer syntax errors. It will come from better logic within executable programs.

Three pieces of this are new. The first is a hierarchical failure taxonomy for program synthesis from images. SpatialTree and PercepTax build tiered spatial taxonomies but score only multiple-choice or text answers, and FractalBench scores programs but only by terminal pixel overlap. The second is a notation manipulation that holds logic content fixed. The third is a perception check that conditions the analysis on inputs the model has already been shown to see correctly.

Research Program

Parameterized ground truth. We re-express each of FractalBench's 12 fractal families, such as Koch curves, Sierpiński gaskets and binary trees, as a parameterized generator. The parameters are the base generator segment sequence, the turn angles, the scale ratio, the branching factor and call pattern, and the recursion depth. Rendering the generator reproduces the benchmark image. If FractalBench's own source programs are released with the benchmark, we use them directly and extract the parameters. If not, we reimplement the families and verify matching renders at the benchmark's IoU of at least 0.95. IoU (intersection over union) measures how much of the two drawings' pixels coincide.

Tiered scoring by execution tracing. Every model program is run under an instrumented turtle that logs the full command trace. The trace is compared with the ground-truth trace at four tiers. Tier 1 checks whether the base generator's shape and segment count are correct. Tier 2 checks angles and length ratios within a tolerance. Tier 3 checks the recursive structure, meaning branching factor and the order of self-calls. Tier 4 checks depth and termination. Syntax validity is recorded separately. Reading tiers from traces rather than parsing source code means any correct program earns credit regardless of style.

Perception check. We borrow the logic of GeoSQA, which swapped diagrams for text descriptions to isolate reasoning. Each image gets short multiple-choice probes asking for the angle, the ratio and the branching factor. Each image is also paired with a condition in which these parameters are stated in text alongside the picture. The probes address the concern raised by Plot Twist, that models can answer harder questions through language shortcuts, because a probe answer counts only if it agrees with the image, not with the prompt. The main analysis conditions on images whose Tier 1 and Tier 2 probes the model answered correctly.

The notation ladder. Five output conditions request the same logical content:

  1. Free Python with standard libraries.
  2. MinimalTurtle Python, as in FractalBench.
  3. MinimalTurtle Python with grammar-constrained decoding. The decoder is allowed to produce only tokens that keep the program syntactically valid.
  4. A block-tree notation. The model emits a nested structure of typed nodes, namely repeat, recurse(depth-1), move(expr), turn(expr) and a base case. This follows the "visible grammar" of Scratch-like editors described by Bau et al., and invalid nesting cannot be expressed. The notation compiles deterministically to turtle code.
  5. A parameter template in which the model only fills in angle, ratio, branching factor and depth.

Condition 5 removes the need to compose recursion, so it is an upper reference for perception plus arithmetic, not a test of interference.

To keep "less syntax" separate from "more structure handed over," conditions 2 through 4 are each run with and without a structural skeleton, which is a recursive function signature with an empty body.

A small 3D extension. Viewpoint robustness is where spatial failures are sharpest: 3DSRBench reports drops of 13.5% for GPT-4o and 32.2% for Gemini-1.5-Pro from common to uncommon camera poses. We therefore add a 3D turtle with pitch and yaw primitives for three families, such as a 3D branching tree, rendered from common and uncommon viewpoints. This asks whether notation effects hold when the model must also undo a camera transform. We treat it as secondary. It is included only if the 2D results show a real effect.

Models. We evaluate the three models FractalBench reports (Gemini 2.5 Flash, Claude 3.7 Sonnet, GPT-4o) through their APIs, using prompting only. For conditions that need control over the decoder, we use open Qwen2.5-VL models, which are also the base for the MVSM model in ViewSpatial-Bench.

Baselines

Naive reference. The do-nothing reference is FractalBench's own protocol: direct MinimalTurtle code generation from the image, scored by IoU of at least 0.95. The strongest reported number is 14.8% accuracy for Gemini 2.5 Flash, against 5.2% for Claude 3.7 Sonnet and 3.5% for GPT-4o (FractalBench). We rerun it under our tiered scorer, so every condition is compared on identical images and identical metrics.

Reasoning before code. The second baseline is natural-language chain-of-thought before code, which means asking the model to write out its reasoning in prose first. FractalBench reports that this "Reasoning-Then-Code" approach degraded accuracy, but gives no per-model figure that we could find. We run it with the same sampling budget and token cap as the ladder conditions.

Structured scratchpad. The third baseline adapts SpatialThinker, whose structured scene tags serve as a spatial scratchpad before answering and improved 3D spatial benchmarks. In our version, the model first writes a structured description of generator, angles and branching, then writes code. This is the most direct competitor to our block notation, because it also adds structure. The difference is that it adds text rather than removing syntax. We find no reported result on Lacuna for a SpatialThinker-style scratchpad on FractalBench.

Execution feedback. The strongest practical competitor is a render-and-revise loop. The model's program is executed, and the rendered image is shown beside the target for up to k revisions. All conditions receive the same total number of generation calls, so feedback cannot win on compute alone. We find no reported result on Lacuna for execution-feedback refinement on FractalBench.

Key ablation. The ablation that isolates our ingredient is grammar-constrained decoding alone, condition 3. It eliminates syntax errors without changing what the model has to write. If the block notation beats condition 3 on Tiers 2 through 4 among executable programs, the gain is interference relief rather than error removal.

What beating the baselines means. We count it as beating them if the block notation improves perception-conditioned Tier 3 accuracy over the best of the other baselines by a margin exceeding the bootstrap 95% interval, on at least two of three model families.

Why not just…?

Why not just use grammar-constrained decoding and call syntax solved? Constrained decoding guarantees valid programs, but FractalBench already shows that validity is not the binding constraint: three quarters of programs run and almost none are correct. Constrained decoding tests whether invalid syntax causes errors. It does not test whether producing syntax at all costs reasoning. Condition 3 is exactly this alternative, and the comparison with condition 4 settles whether it suffices. If the two match, the interference hypothesis fails, and we will report that.

Why not just ask for the parameters and skip programs entirely? A fill-in template (condition 5) may well score highest. That is a real possibility, not a straw man. However, it hands the model the recursive structure, which FractalBench identifies as the main failure: models score about 20% on linear families but under 2% on branching ones. It also does not extend to scenes without a known parametric family. We report it as the ceiling for perception plus arithmetic. If the block notation with a skeleton approaches that ceiling, composition is cheap once syntax is removed. If it stays far below, composition itself is the problem.

Why not just add execution feedback? Feedback may beat everything else, and it is the most serious threat to the practical value of this work. It does not answer the scientific question, because it corrects errors after the fact rather than revealing where they arise. We therefore run feedback crossed with notation. If feedback's gain is larger in text notations than in the block notation, feedback is partly compensating for interference. If its gain is the same in both, the two effects are independent.

Why not just reason in prose first, or use a bigger model? FractalBench reports that prose reasoning before code hurts. MATH-Vision finds chain-of-thought inconsistent for multimodal models. SpatialTree finds that verbose thinking degrades low-level perception. Our tiered scorer can show which tier prose reasoning damages, for instance whether Tier 2 angles drift. On scale, Vision Language Models See What You Want but not What You See and MARBLE show that spatial transformation abilities barely move with size. We test scale directly by comparing Qwen2.5-VL sizes across the ladder.

Decisive Checks

The central check is a perception-conditioned comparison. On images where the model answers the angle and branching probes correctly, and restricting attention to programs that execute, we compare Tier 3 and Tier 4 accuracy for block notation against constrained Python. Both are run with the same skeleton setting and the same sampling budget. The test is a paired bootstrap over images.

A positive result would show three things together. The block notation would win within executable programs. The win would be concentrated in branching families, where FractalBench finds the collapse. And giving the parameters in text would narrow but not close the gap, which would mean the remaining failure lies in composition rather than perception.

A negative result would also be informative. If the gains disappear once perception is correct and code is valid, then syntax-logic interference is a human-cognition story that does not carry over to these models. The tiered trace scorer would still be released as a diagnostic that FractalBench-style benchmarks currently lack.

The 3D viewpoint extension asks one further question: whether the notation effect grows under uncommon camera poses, as the 3DSRBench degradations would suggest.