Vision-language models (VLMs) answer questions about images by cutting each image into small patches and feeding those patches to a language model as one long row of tokens. The row is usually built in raster order: top-left patch first, bottom-right patch last. Text models are known to over-trust whatever comes first in their input, a habit called lead bias. Raster order therefore turns the top-left of every image into the "first sentences" of a document. Most VLM decoders also use causal attention, meaning each token can only look at tokens before it. Under that rule, early patches never see later patches inside the language model, while late patches see everything. We propose to test whether this reading order produces a measurable, directional bias in spatial reasoning. The concrete prediction is that questions about objects rendered late in the token sequence are answered less reliably than questions about the same objects rendered early. If the bias exists, it should be removable by a cheap change to how patches are ordered during fine-tuning, without new spatial data.
This matters because the spatial-reasoning literature attributes VLM failures almost entirely to missing geometric knowledge: sparse spatial language in captions, no 3D world model, and confusion about viewpoint. None of those accounts predicts that accuracy should depend on where in the token stream an object happens to fall. A clean positive result would show that some of the measured spatial deficit is a positional shortcut sitting on top of the geometric one. That would change how spatial benchmarks are read and which fixes are worth pursuing. A clean negative result would also be useful, because it would rule out a mechanism that several recent papers implicitly worry about.
Core Claim
We claim that, in raster-flattened VLMs with causal decoders, spatial question accuracy depends on an object's sequence position independently of its physical image location. We further claim that fine-tuning with randomized patch reading order, while each patch keeps its true 2D position code, removes most of this dependence and improves controlled spatial benchmarks more than the positional-embedding redesign that currently targets related biases.
The key methodological move is a factorial design that pulls apart two things raster order normally ties together: where an object is in the picture, and where its tokens land in the sequence. Flipping the image moves the object physically. Reversing the scan order independently moves its tokens in the sequence. Crossing the two gives a 2×2 design. A lead-bias effect shows up as a main effect of sequence position that persists when physical location is held fixed.
Relation to Prior Work
Work that motivates the question comes from both directions. Collapse of Dense Retrievers stress-tests dual-encoder retrievers. It shows that they pick short, early-keyword foils over real evidence more than 90% of the time, and that this cuts downstream GPT-4o accuracy by 34%. The lesson we take is that selection-style decisions are especially vulnerable to early position cues, and "which object is left of the cup" is exactly such a selection.
Against this, Not Lost After All uses cross-encoder attribution to argue that apparent lead bias in LLM summarization is partly an artifact of n-gram metrics. It reports a rightward shift and 90% positional stability under sentence permutation. This constrains our design. We measure behavior under controlled interventions, not attention maps or overlap scores, so a null result is informative rather than an artifact.
On the spatial side, three benchmarks document the deficit and supply our test beds. What's "up" builds tripod-controlled scenes where only the spatial relation changes. Visual Spatial Reasoning covers 66 relations in natural images. ViewSpatial-Bench tests camera-centric versus person-centric viewpoints. All three explain failures through data or geometry and never vary token order.
The closest existing work is Circle-RoPE. RoPE, or rotary position embedding, encodes each token's position as a rotation. Circle-RoPE observes that text tokens attend to image tokens partly by index proximity. It fixes this by placing image positions on a cone so that every text token is equidistant from every image token. Our target is different in two ways. Circle-RoPE addresses text-to-image distance and leaves the raster order, and the causal mask among image tokens, untouched. That order and mask are where a top-left-favoring asymmetry would originate. Circle-RoPE was also evaluated only on general benchmarks such as MMMU, never on What'sUp, VSR, or ViewSpatial-Bench.
Early Decisions Matter shows that the order in which diffusion language models commit to tokens shapes their reasoning. It also reports that its fix breaks down on Sudoku, a 2D grid task. This is a caution that order interventions on grid structure can disrupt what a model has learned, and it is why we fine-tune rather than only perturb at test time. Finally, SpatialThinker and 3ViewSense improve spatial reasoning through reward-shaped or structured reasoning traces. We treat them as complementary: our change is in how the image is read, not how the answer is reasoned out.
Research Program
The first stage is a diagnostic benchmark built from existing data. From What'sUp and VSR, we create four variants of each item. The first is the original image with the original question. The second is the image flipped with its left/right or up/down labels remapped. The third is the original image with the patch sequence reversed; each patch keeps its original 2D position code, so the model is told the true geometry but reads it in a different order. The fourth is the flipped image with the reversed sequence.
For models that assign explicit 2D position codes to patches, as Qwen2.5-VL does, reordering the sequence while preserving the codes is well defined. For LLaVA-style models with 1D positions, the codes necessarily move with the tokens. Comparing the two families separates the effect of the causal mask from the effect of positional encoding. We add a synthetic "transplant" set with the same object pair pasted at controlled grid locations, which gives a continuous measure of sequence position. On all of these we report accuracy as a function of the target object's normalized sequence position.
The second stage is mechanistic. We measure how much attention answer tokens pay to object patches as a function of sequence index. We then test a single change: allowing bidirectional attention among image tokens only, with no training. This asks whether the asymmetry comes from the causal mask. We expect this zero-shot change to hurt somewhat, as the Sudoku result warns, so it serves as a probe rather than a fix.
The third stage is the fix. We fine-tune Qwen2.5-VL-3B with each image's patches presented in a random order drawn per example, still carrying true 2D position codes. We use a general instruction mixture that contains no added spatial data. The model therefore cannot rely on scan position and must rely on the position codes. A second variant adds bidirectional attention within the image block during fine-tuning. Both are compared against Circle-RoPE trained on the same mixture.
Baselines
The naive reference point is the unmodified base model evaluated in its native raster order. For context, the best reported model on What'sUp, BLIP fine-tuned on VQAv2, reaches 56% against 99% for humans (What's "up"). On VSR, reported models plateau around 70% against 95.4% for humans (Visual Spatial Reasoning).
A second baseline is plain fine-tuning in raster order on the identical instruction mixture, with the same steps, learning rate sweep, and seeds. This isolates the reordering itself from the effect of any additional training.
The strongest competitor that targets positional bias is Circle-RoPE, because it is the only method in the file that modifies positional structure to reduce cross-modal bias. It reports +1.00 on MMMU validation over a Qwen2.5-VL fine-tuning baseline. We found no reported Circle-RoPE result on What'sUp, VSR, or ViewSpatial-Bench on Lacuna. We re-implement it on the same backbone, data, and compute, using the authors' combination of cone-shaped and standard encodings across layers.
The strongest spatial-specific system is MVSM, which reaches 82.09% on ViewSpatial-Bench, an absolute gain of 46.24 points over base Qwen2.5-VL-3B (ViewSpatial-Bench). MVSM uses 43,000 targeted perspective examples, so we do not expect randomized reading order alone to match it. The fair comparison is to add randomized order on top of MVSM-style training with the same data. Beating that combination on ViewSpatial-Bench, or on the positional gap metric, is our bar.
Ablations isolate the key ingredient. We run randomized order without bidirectional attention, and bidirectional attention without randomized order. We also run a flip-augmentation baseline, described in the next section.
Winning has two parts. The first is a significant reduction in the accuracy gap between early- and late-sequence targets relative to raster fine-tuning, with a pre-set threshold of halving it. The second is a gain on What'sUp and VSR over both raster fine-tuning and Circle-RoPE, with no more than a one-point loss on MMMU and RealWorldQA.
Why not just…?
Why not just scale the model? Vision Language Models See What You Want but not What You See reports essentially no gain in perspective-taking from scale, and GPT-4o scores 34.98% on ViewSpatial-Bench. We also run the diagnostic on the largest open model we can afford. A lead-bias effect that shrinks with scale would itself be a finding worth reporting.
Why not just train with image flips and remapped labels? This is a real threat. Flipping does spread each object across both ends of the sequence, so it could erase the positional gap without touching reading order. It has two weaknesses. It corrects only the flipped axis, and it is unsafe for relations that do not flip, such as text or intrinsically oriented objects; VSR notes that relations like "facing" are already hard. We run flip augmentation at matched compute. If it matches randomized order on the gap metric and on the benchmarks, we will report that the cheaper fix suffices, and the diagnostic contribution stands on its own.
Why not just use Circle-RoPE? Circle-RoPE equalizes text-to-image distances but leaves the order of image tokens and the causal mask among them intact. If our hypothesis is right, it should leave a residual gap between early and late targets. The head-to-head comparison on the gap metric tests this directly.
Why not just fine-tune on spatial QA? Kamath et al. found that spatial fine-tuning often lowered What'sUp accuracy, because models picked up dataset-specific shortcuts (What's "up"). Our fix deliberately adds no spatial data, so any gain cannot come from that route.
Decisive Checks
The central test is the 2×2 factorial on What'sUp and VSR. If lead bias is real, accuracy should follow sequence position when physical location is fixed. Concretely, the original image read in reversed order should behave like the flipped image read in raster order, for targets that land in the same sequence positions. If instead accuracy follows only physical location or linguistic priors such as the reported on-table bias, the hypothesis is falsified for that model family.
Next, the effect should be larger in causal-decoder models than in models whose image tokens already attend bidirectionally. The gap should also shrink under bidirectional image attention, which would indicate the causal mask as the mechanism.
Finally, randomized-order fine-tuning should close the gap and improve controlled spatial accuracy, with no loss on general benchmarks. If the diagnostic gap is under two points, we will report it as a bounded negative result. That would favor the geometric-deficit account and support the measurement-artifact caution of Not Lost After All.
References
- Collapse of Dense Retrievers: Short, Early, and Literal Biases Outranking Factual Evidence
- Not Lost After All: How Cross-Encoder Attribution Challenges Position Bias Assumptions in LLM Summarization
- What's "up" with vision-language models? Investigating their struggle with spatial reasoning
- Visual Spatial Reasoning
- ViewSpatial-Bench: Evaluating Multi-perspective Spatial Localization in Vision-Language Models
- Circle-RoPE: Cone-like Decoupled Rotary Positional Embedding for Vision-Language Models
- Early Decisions Matter: Proximity Bias and Initial Trajectory Shaping in Non-Autoregressive Diffusion Language Models
- SpatialThinker: Reinforcing 3D Reasoning in Multimodal LLMs via Spatial Rewards
- 3ViewSense: Spatial and Mental Perspective Reasoning from Orthographic Views in Vision-Language Models
- Vision Language Models See What You Want but not What You See