Where Vision Stops Being Read: Allocating Depth for Multimodal Reasoning

Vision-language models can name every object in a geometry diagram and still get the problem wrong. On the MATH-Vision benchmark, most of GPT-4V's failures are reasoning errors in which the model perceived the diagram correctly, not perception errors. A separate body of work on text-only language models finds that only about the first half of a network's layers does new computation. The later layers mostly polish an answer that is already formed. This note joins the two findings. We propose that in multimodal models, the image is read only during that early working phase. Multi-step problems fail when a later step needs a visual fact that the network no longer consults.

The plan has two parts. First, we measure a "visual cutoff depth": the layer after which blocking the model's access to image tokens no longer changes its answers. We then test whether this cutoff sits at the same fixed fraction of depth that text-only models show, regardless of model size or problem difficulty. Second, we try to move the cutoff without adding inference tokens. We insert a few new residual layers, initialized so the model's outputs are initially unchanged, at positions chosen by a gradient diagnostic computed on image-conditioned loss. We then fine-tune briefly. The question is whether placing new depth before the cutoff closes part of the reasoning gap where the same added depth elsewhere does not.

Core Claim

Our hypothesis is that the multimodal reasoning gap is partly a depth-allocation problem. The fraction of layers that both read visual tokens and perform composition is small and fixed, and the extra layers in a deep model mostly average rather than add visual-relational steps.

Two prior findings motivate this. What Affects the Effective Depth of Large Language Models? uses logit-lens decoding, layer skipping and residual similarity to show that in Qwen-2.5 and DeepSeek-R1-Distill text models, the share of layers doing composition stays between 0.60 and 0.70. It does not grow with scale, with long chain-of-thought training, or with task difficulty. UAOR shows that in vision-language-action robot models, attention to visual tokens decays sharply in intermediate and deep layers.

Neither paper connects the two observations, and neither studies reasoning benchmarks. If the cutoff and the composition-to-polishing transition coincide, three things follow:

  • Simply making the language backbone deeper should help visual reasoning little, as Inverse Depth Scaling From Most Layers Being Similar would predict. That paper finds loss falls roughly as depth to the power −1.1, the signature of layers averaging out errors rather than assembling new computation.
  • Depth added inside the visual-reading window should help disproportionately on problems that need several visual facts.
  • That gain should vanish when the same problem is supplied as a text description.

Research Program

Measuring the cutoff. For a layer index $k$, we mask attention from all text and generated positions to image-token positions in every layer at or above $k$. We record the accuracy $A(k)$, normalized between the fully masked case and the unmasked case. We define the visual cutoff depth $k^*$ as the smallest $k$ at which normalized accuracy reaches 0.95. Above this layer, the image no longer matters.

We compute $k^*/L$, where $L$ is total depth, for the Qwen2.5-VL language backbone at 3B, 7B and larger sizes, plus InternVL and the reasoning-tuned VLAA-Thinker-Qwen2.5VL-3B. We compute it separately on MATH-Vision difficulty levels, Chart101 elementary probes and DisTANCE. Alongside, we compute the composition/polishing transition with the same tools as Zhou et al., applied to the same image-conditioned inputs.

The first finding is whether $k^*$ and the transition coincide, and whether $k^*/L$ is invariant the way effective depth is. We also apply the cutoff measurement to answers generated with chain-of-thought. This tests whether each generated reasoning token re-reads the image only in early layers, which would explain why chain-of-thought helps unreliably on MATH-Vision.

Choosing where to add depth. Quantifying Depth Sufficiency in Residual Neural Networks supplies the method. It shows that a network is saturated under first-order updates exactly when the loss gradient at candidate insertion points has no component that a new residual block could represent. The maximum of that quantity over insertion points, called $AG_{max}$, plateaus at the depth where test loss stops improving. That paper validated the criterion only on unimodal ResNets and GPT-2/Pythia.

We compute it per insertion point $j$ on the loss of image-conditioned reasoning data, and split it by whether the residual block can attend to image tokens:

$$ AG(j) = \left\| P_j \, \nabla_{h_j} \mathcal{L}_{\text{mm}} \right\| $$

Here $h_j$ is the residual stream after layer $j$, $\mathcal{L}_{\text{mm}}$ is the next-token loss on multimodal reasoning traces, and $P_j$ projects onto the functions a freshly inserted attention-plus-MLP block at $j$ can represent. We compare $AG(j)$ with image access allowed and with it blocked. The gap between the two measures how much useful new computation at $j$ needs to read the image.

We insert a small number of new blocks, for example four, at the positions with the largest gap. Each block's output projection is zero-initialized, so insertion preserves the model's function. We then fine-tune on a fixed multimodal reasoning mixture: VeriSciQA plus MATH-Vision-style training data excluding test items.

After training, we re-measure $k^*$. We also check whether the inserted blocks form the internally reorderable "computational blocks" that Do Depth-Grown Models Overcome the Curse of Depth? observed when text models are grown from the middle. That paper's 23% FLOP savings were shown only for text math. The comparison tells us whether middle-growth's benefits transfer to the cross-modal setting or whether visual-gradient placement matters beyond "insert in the middle."

Prior work this differs from. Stacked Latent Attention for Multimodal Reasoning fixed a depth plateau at two attention layers in convolutional/LSTM VQA systems by moving softmax off the main path. It scaled only to three steps and was evaluated on VQA 2.0, so it says nothing about 30-plus-layer transformers. UAOR re-injects visual features into later layers at test time when action uncertainty is high. It treats depth as fixed and targets robot control; we change where depth sits and evaluate on mathematical and chart reasoning. Perception-Time Scaling raises late-layer visual attention by making the model emit textual rulers and decomposition tokens. It buys its gains with longer outputs, whereas we spend parameters inside one forward pass and compare at matched compute.

What constrains the claim. GeoSQA found that replacing maps with human-written descriptions left models near chance. This warns that some of the gap is language-side reasoning weakness that no visual re-reading can fix. The text-description control below is designed around exactly that finding.

Baselines

Unmodified backbone. The naive reference is Qwen2.5-VL-7B (and 3B) as released, plus the same model fine-tuned on our training mixture for the same number of steps with no inserted layers. The second version is the real control: any gain from insertion must exceed the gain from fine-tuning alone. The file reports no MATH-Vision number for Qwen2.5-VL ("no reported result found on Lacuna for Qwen2.5-VL on MATH-Vision"), so we measure it ourselves. For context, the MATH-Vision paper reports GPT-4o at 30.39%, the best open model at the time (InternVL-Chat-V1-2-Plus) at 16.97%, and humans at 68.82% (MATH-Vision).

Perception-Time Scaling. This is the strongest existing method on precise visual quantities. Its paper reports that Qwen2.5-VL-7B on DisTANCE improves from 8.0% to 64.7% in high-precision relative accuracy ($RA_{0.1}$), and from 21.5% to 88.3% in average relative accuracy. We reproduce PTS-style fine-tuning on the same backbone with the same tuning budget. We report two settings. In equal training compute, PTS gets the same fine-tuning steps. In equal inference FLOPs, our model's extra layers are charged against PTS's extra output tokens. Beating PTS on DisTANCE is not expected. The bar is to match or beat it on MATH-Vision and Chart101 at equal inference FLOPs, and to show that combining the two is additive.

UAOR-style reinjection. We adapt UAOR's uncertainty-triggered retrieval of visual embeddings into later feed-forward layers. Language-model output entropy replaces action entropy as the trigger, and the threshold is tuned on a held-out split. This is the strongest test-time competitor for "late layers forget the image."

Ablations that isolate placement. We train the same number of zero-initialized blocks under four placements, all with identical data and steps:

  • inserted after the cutoff (in the polishing layers);
  • inserted uniformly across depth;
  • inserted at the exact middle, following the LIDAS-style middle-growth rule;
  • selected by text-only $AG_{max}$ rather than the image-conditioned gap.

The proposal's key ingredient is placement guided by the image gradient. We claim it works only if the image-gradient placement beats all four by a margin larger than seed variance (three seeds each) on the vision-dependent subsets.

Why not just…?

Why not just generate more reasoning tokens (chain-of-thought or PTS)? Token scaling is a real competitor, and on metric estimation it will probably win. The MATH-Vision authors report that chain-of-thought is inconsistent on their benchmark and often hurts. Zhou et al. show that long chain-of-thought training does not change how many layers do composition per token. If our cutoff measurement shows that each generated token still reads the image only in early layers, that explains why more tokens help unevenly. The matched-inference-FLOP comparison and the PTS-plus-insertion combination settle whether internal depth adds anything token scaling does not.

Why not just re-inject image features into every layer, or use UAOR? UAOR itself reports that indiscriminate reinjection into every layer floods intermediate representations and does worse than selective reinjection. Even selective reinjection only hands the late layers visual features. If those layers are in a polishing phase, they may lack the capacity to compose with them. We run UAOR-style reinjection as a baseline. If it equals our method, the "fixed composition window" story is wrong and the problem is simply forgetting; we would report that outcome.

Why not just use a bigger vision encoder or higher resolution? The MATH-Vision error analysis attributes 42.2% of GPT-4V failures to reasoning with correct perception, against 31.9% to recognition. Our text-description control directly tests the encoder explanation. If gains from inserted depth appear when the diagram is given as an image but not when the same content is given as text, the bottleneck lies in cross-modal composition, not pixels.

Why not just make the backbone deeper or wider? Inverse depth scaling predicts that uniformly added layers mainly average errors away. Wide residual networks showed that width can substitute for depth in vision models. Our uniform-placement ablation tests the "just deeper" version at matched parameters. We also add a width control: equal parameters added by widening the MLP of the same layers.

Proposed Experiments

The first decisive check is descriptive. We test whether $k^*/L$ for Qwen2.5-VL and InternVL falls within the 0.60–0.70 band Zhou et al. report for text models. We also test whether it stays fixed across MATH-Vision's five difficulty levels and across model sizes. Constancy supports a fixed window. A cutoff that moves deeper on harder problems would undercut the allocation story.

The second check uses Chart101. That benchmark's authors found sub-25% collective accuracy and 4.2% on unlabeled scatter plots (Plot Twist), plus a heavy reliance on chart titles. We compute $k^*$ separately with and without titles. A shortcut-driven answer should show a shallow cutoff, with the image ignored early. A vision-driven answer should show a deeper one.

The third check is the intervention. Accuracy is measured on MATH-Vision overall and on its Transformation Geometry and Arithmetic splits, Chart101 elementary probes, DisTANCE, and OlympiadBench multimodal items. All placements and baselines above are compared at matched training and inference compute, with three seeds.

The fourth check is the text-description control. On a MATH-Vision subset where we write a full textual description of each diagram, the image-gradient-placed blocks should give their gains in the image condition and little in the text condition. Uniform or text-placed blocks should not show this asymmetry. We also re-label errors on 200 MATH-Vision failures using the original recognition-versus-reasoning taxonomy, to see which category shrinks.

The proposal fails cleanly if any of three things holds: $k^*$ does not align with the composition transition, image-gradient placement is no better than middle or uniform insertion, or UAOR-style reinjection matches it. Each outcome would still be informative about where multimodal models stop using the image.