Spatial or Spelling? Separating the Causes of Grid-Reading Failure in Vision-Language Models

Vision-language models (VLMs, models that answer questions about images in text) do badly on tasks laid out as letter grids: crosswords, word squares, simple floor-plan maps. The usual explanation is that they are "spatially blind." There is a second explanation that is just as plausible. Language models read text in sub-word chunks called tokens, not in single characters, so they cannot reliably say which letter sits at a given position. A grid needs both skills at once: finding the right cell, and reading the exact character in it. When a model misreads the vertical word in a crossword, nobody currently knows whether it failed to move down a column, failed to isolate a single character, or simply guessed a familiar English word that fit.

This proposal builds a procedurally generated grid benchmark that separates those three causes. It crosses four factors: the format the grid arrives in (image versus several text layouts), the direction of reading (across, down, diagonal), whether the letters spell real words or random strings, and a three-level ladder of tasks running from reading one cell, to relating two cells, to filling a whole grid under exact character constraints. Because every instance is generated fresh with a known answer, scores cannot come from memorized puzzles. The payoff is a per-model diagnosis of why grid tasks fail, which we then check causally: each diagnosed cause should predict which cheap fix helps that model, and which does not.

Core Claim

The clearest existing evidence is CrossWordBench. It generates crosswords to avoid contamination and presents them as text or as rendered images. It reports that VLM puzzle accuracy tracks grid-parsing accuracy with correlation $r = 0.9358$, and that VLMs extract Across words much more accurately than Down words. That correlation names the bottleneck but does not explain it. A Down-word failure could be visual, if the image encoder handles vertical layout poorly. It could come from tokenization, the token-perception gap documented for length control in MarkerGen. Or it could be lexical, with the model reconstructing a plausible word from its letters instead of reading them.

Our claim is that these causes can be measured separately, and that they differ across model families. The prediction we will test is concrete. In text-only grids, much of the Across/Down asymmetry should disappear when each cell becomes its own token with an explicit coordinate. That would be a spelling problem. In image grids, the asymmetry should survive any text-side change but should flip when the image is rotated 90 degrees. That would be a reading-direction bias in the visual pathway, not a failure of spatial reasoning. If neither prediction holds, and the deficit persists for random-letter grids under every format, then the "spatial blindness" reading is right for these tasks, and we will have shown it more cleanly than before.

Research Program

The generator produces grids from 5×5 to 15×15. Each grid is filled either with dictionary words or with letter strings sampled to match English letter frequencies but forming no words. The nonword condition is the contamination and shortcut control. It removes the lexical prior that Benchmark Designers Should "Train on the Test Set" shows inflates spatial benchmarks. That paper found blind models gaining more than 30 points from text priors alone.

Each grid is rendered in five ways: as an image, and as four text layouts. The text layouts are plain rows, space-separated characters, column-major order, and coordinate-tagged cells such as "r3c4=K". The layouts are chosen so that each one changes exactly one thing. Space separation forces one character per token. Column-major order turns down-reading into across-reading in the text stream. Coordinate tags remove the need to count positions.

The task ladder borrows the tier structure of SpatialTree, which organizes spatial skills as perception, mental mapping, simulation and action on static scene data. We adapt it to discrete grids.

At level 1 the model reads one cell, one full row, or one full column.

At level 2 it answers relational queries. Examples are "which letter is two cells left of the only Q?", "do row 3 and column 5 share a letter at their crossing?", and "after a 90-degree rotation, which letter is at r1c1?"

At level 3 it must compose. It fills a partially blanked grid so that every across and down string matches a provided list, or writes out a transformed grid. The output must have exactly the right character count in every row. That last requirement brings in character-level constraint adherence as a measured output property. We score each output for row-length violations separately from content errors, following the observation in WildIFEval that exact length bounds are routinely broken.

Every level-3 item is paired with the level-1 and level-2 probes on the same grid. This lets us ask whether a composition failure is predicted by a failure lower down. The tier structure is therefore a set of conditional measurements, not just a reporting format. It also addresses the "task-tier entanglement" problem noted for hierarchical benchmarks: the lower-tier probes use the same grid and the same format as the composition task.

The analysis fits, for each model, a simple logistic model of cell-level correctness. The inputs are format, reading direction, word versus nonword, tier, and their interactions. Each diagnosed cause corresponds to a specific interaction:

  • Tokenization: the direction × space-separated term.
  • Visual reading direction: the direction × rotation term in images.
  • Lexical prior: the word × nonword gap.

The causal check follows. For each model we apply the fix matching its largest diagnosed cause and the fix matching its smallest. The fixes are character-per-token serialization, image rotation or cropping to single columns, and nonword-trained prompting examples. If the diagnosis is right, the matched fix should beat the mismatched one.

Baselines

The naive reference points are chance accuracy for each probe and a blind baseline. The blind baseline follows the "train on the test set" procedure of Benchmark Designers Should "Train on the Test Set": a text-only model is fit on the question strings without the grid, so we can measure how much each item can be answered from priors. Any tier where the blind baseline comes within a few points of real models is redesigned before results are reported.

The strongest existing system on the underlying task is a text reasoning model solving crosswords. CrossWordBench reports o3-mini at intersection consistency rate (ICR) 0.891 and word completion rate (WCR) 0.587 on 7×7 grids. It reports Claude 3.7 Sonnet, the strongest VLM there, at WCR 0.479. We will rerun these model classes on our generated 7×7 grids, with the original CrossWordBench items from HINT-lab/CrossWordBench as an anchor. All models get the same prompts, the same decoding settings, and the same output-token budget per item. This tells us whether our generated items behave like theirs.

The strongest existing diagnostic method is CrossWordBench's own analysis: correlating grid-parsing accuracy with solve accuracy. Our approach beats it if the factorial attribution predicts which intervention helps a given model better than the parsing-accuracy score alone. Concretely, we fit both predictors on half the models and measure on the other half how well each predicts the gain from each fix. A diagnostic that does not improve that prediction has added nothing.

The key ablations remove one ingredient each. The first collapses tiers into a single flat solve score. The second drops the nonword condition. The third drops the alternative text layouts. Each should lose the ability to separate at least one cause.

Why not just…?

Why not just use CrossWordBench's grid-parsing correlation, since it already locates the bottleneck? A correlation of 0.9358 between parsing and solving says that perception limits solving. It does not say which part of perception. Our Across/Down × layout × rotation design is exactly the experiment that splits it. If the attribution adds no predictive value over parsing accuracy in the held-out intervention test above, we will report that the simpler measure suffices.

Why not just give the model a text transcription or an OCR tool? This is a real threat. If coordinate-tagged text closes nearly all of the image deficit for every model, then grid failure in VLMs is an image-reading problem, not spatial reasoning, and the practical answer is "transcribe first." The proposal treats that as a possible headline result, not something to argue against. The text-only condition is not trivially solved, though. CrossWordBench shows text-mode non-reasoning models also fall well short of o3-mini. Our layout factor tests whether that remaining text gap comes from character tokenization.

Why not just let the model think longer? SpatialTree reports that encouraging extended reasoning improved simulation-level tasks while harming basic perception. Time's Up! shows reasoning models waste tight budgets inside their thinking blocks. We include a sweep over output budgets at each tier. Our prediction is that longer thinking helps level-3 constraint satisfaction on text grids but not level-1 reading of image columns. If extra thinking also fixes level-1 image reading, the perceptual account is weakened.

Why not just use an existing spatial hierarchy such as SpatialTree or a dynamic geometry benchmark? SpatialTree is assembled from static scene datasets, which are vulnerable to the shortcut leaks documented for VSI-Bench in Hidden Sensitivity in Spatial Reasoning Evaluation. DynaSolidGeo perturbs geometry problems to expose a memorization tax, including a 20.4% drop for Claude-Sonnet-4.5. Neither manipulates the character level, so neither can tell tokenization apart from spatial layout.

Proposed Experiments

The main study runs open VLMs (Qwen2.5-VL, InternVL2.5, Qwen3VL), proprietary VLMs (GPT-4o, Claude 3.7 Sonnet, Gemini), and text reasoning models (o3-mini, DeepSeek-R1, text conditions only). It uses roughly 2,000 generated grids per size, each with all tier probes and all renderings. Metrics are:

  • cell-level and word-level accuracy per tier;
  • CrossWordBench's ICR and WCR for level-3 fills;
  • exact row-length adherence;
  • the Across-minus-Down gap under each format.

There are three decisive checks.

First, image rotation should move the Across/Down gap, while text layout changes should not, for models diagnosed with a visual cause. For text-only models, space-separated or coordinate-tagged layouts should shrink the gap.

Second, the word-minus-nonword gap measures how much level-3 success comes from guessing familiar words. If that gap exceeds the tokenization gap, lexical shortcuts dominate. If it is near zero, the prior is not driving scores.

Third, the matched intervention for each model should beat its mismatched intervention more often than chance across held-out models. This is the test that tells us whether the tiered, factorial diagnosis explains failures or only describes them.

For external validity, we test whether level-1 column-reading accuracy on our nonword image grids predicts each model's Down-word accuracy on the original CrossWordBench items. If it does not, our probes measure something other than what makes real crosswords hard, and we will say so.