Vision-language models often answer hard chart and geometry questions correctly while failing simple ones, such as reading an axis limit or a point's coordinate. A common diagnosis is that the image encoder squeezes away fine spatial detail before the language model ever sees it. Separately, work on very deep transformers has shown that changes to the residual and normalization wiring can keep 128- to 512-layer stacks trainable and expressive. Here a residual connection is the shortcut that adds a layer's input back to its output, and normalization rescales features so they stay in a stable range. This proposal joins the two lines with a controlled experiment. We fix the total parameter and compute budget of a vision-language model and move depth among three places: the image encoder itself, a small "pre-aligner" network between encoder and language model, and the language model. The test is whether depth placed inside the encoder, with stabilizing wiring, specifically repairs perceptual failures. We expect depth placed downstream to help reasoning failures instead.
The central bet is a crossover. A deep, narrow image encoder built with spanning residuals should keep fine spatial detail all the way to its final layer. Spanning residuals add a whole block's input to its output and then re-normalize. If that detail survives, the encoder should close much of the gap on elementary chart reading and on vision-attributed math errors, while doing little for errors that persist when perception is perfect. If instead the same parameters do equally well as a wider encoder that reads out several intermediate layers, then encoder depth is not the lever, and the field should stop pursuing it.
Core Claim
The Perception Encoder study (Perception Encoder: The best visual embeddings are not at the output of the network) found a "dichotomy of layers" in a 50-layer contrastive encoder. Spatial tasks peak near layer 32, semantic tasks near layer 47, and the final layer is degraded by the global contrastive objective. That study fixed the problem with two separate alignment-tuning recipes. We claim that part of this mid-network peak is an artifact of the standard residual wiring, called PreNorm, in which normalization is applied before each sub-layer. Our reasoning comes from SpanNorm: Reconciling Training Stability and Performance in Deep Transformers, which shows that in language models PreNorm layers drift toward identity mappings with depth. SpanNorm adds three changes: a block-spanning residual, a variance reset after the block, and output-weight scaling of $1/L$ for an $L$-layer network. Together these roughly double the effective dimension ratio, from 12.99 to 30.66, and scale monotonically to 512 layers. SpanNorm has only been tested on text (SlimPajama). Our first prediction is that a SpanNorm image encoder shows a much smaller gap between its best intermediate layer and its final layer on spatial probes than a PreNorm encoder of the same size. If so, a single output feed can carry both kinds of information.
The second claim concerns where the gains land. Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset attributes 31.9% of GPT-4V's failures to visual recognition and 42.2% to reasoning. GeoSQA shows that even gold text descriptions of diagrams leave older text models near chance: 26.22% against 25% random. That result constrains the proposal, because encoder improvements cannot be expected to fix reasoning. We therefore predict that encoder depth reduces perception-attributed errors and leaves reasoning-attributed errors flat. Spending the same parameters on a pre-aligner, as in Deep Pre-Alignment for VLMs, should show the opposite pattern. The pre-aligner targets the language model's early layers being spent on modality conversion. The two claims are linked: if final-layer spatial detail is not preserved, the crossover should not appear.
Research Program
We build a family of image encoders at a fixed parameter count of roughly 400M. Depths are 24, 48, 96 and 192 layers, with width shrinking to hold parameters constant. Each depth is trained in two wirings, PreNorm and SpanNorm with Scale Init. Every arm shares the same contrastive image-text pretraining data, augmented with synthetic charts and diagrams, the same objective, the same token budget, and the same learning-rate search. Holding the objective fixed matters. If the contrastive objective is the main source of final-layer damage, as Perception Encoder suggests, we will see it in all arms equally and can separate it from the effect of wiring.
We then analyze each encoder layer by layer. Linear probes measure point and coordinate localization on rendered plots with known ground truth, box localization on COCO, and semantic classification. For each layer we also report the effective-dimension statistic used by SpanNorm. The quantity of interest is the final-minus-peak gap on spatial probes.
Each encoder is attached to the same Qwen3-4B language model, the backbone used by DPA, and trained with the same projector and multimodal instruction data. Within one fixed total budget, we compare three variants. In the first, depth is placed in the encoder. In the second, a shallower encoder is paired with a DPA-style perceiver of matched parameters. In the third, a shallower encoder is paired with extra adapter layers in the language model. We follow DPA in keeping the encoder instruction-agnostic, because DPA reports that feeding the question into the perceiver hurt multi-turn use.
Baselines
The naive reference point is a standard PreNorm vision transformer of the same parameter count and conventional depth (24 layers), feeding its last layer to the language model. Every other arm must beat this to matter. It runs on identical data, steps and tuning budget.
The strongest competitor on the spatial-detail question is multi-layer readout from that same standard encoder. Here we concatenate or learn a weighted sum of the layers at roughly two-thirds depth and at the output, in the spirit of Perception Encoder's use of intermediate layers. It is the strongest competitor because it targets exactly the lost-detail mechanism, and it does so without re-architecting. For scale, Perception Encoder reports 94.6 on DocVQA and 80.9 on InfographicVQA for PE-lang-G with an 8B language model, and 66.0 box mAP on COCO for PE-spatial-G. Our models are far smaller, so these figures are context rather than targets. Beating this baseline means the deep SpanNorm encoder, read only at its final layer, matches or exceeds multi-layer readout on Chart101-style elementary reading and on our coordinate probes, at equal parameters and readout token count.
The strongest competitor on the downstream-reasoning question is DPA's perceiver. DPA reports an average gain of 1.9 points over its 4B baseline across eight benchmarks including MathVision and MathVista, with no per-benchmark MathVision number reported. It also reports MATH-500 retention of 54.2 against 36.4 for the standard baseline, from a base of 84.8. We reimplement DPA at matched parameters, on our data, with the same tuning sweep. Beating it on perception means fewer perception-attributed errors and better Chart101 robustness. We do not expect, or require, a win on reasoning-attributed errors.
The ablation that isolates the key ingredient is the PreNorm encoder at matched depth and width. If deep PreNorm captures most of SpanNorm's gain, then depth matters and the wiring does not. A second ablation removes Scale Init, which tests the file's noted dependence on initialization.
Why not just…?
Why not just tap intermediate layers of an existing encoder? This is the real threat, and the design treats it as the main baseline rather than arguing it away. Perception Encoder needed separate spatial and language alignment and did not obtain one feed serving both, which suggests readout alone leaves something on the table. Still, a learned multi-layer readout is cheap and could match our results. If it equals the deep SpanNorm encoder on elementary chart reading and on vision-attributed MATH-Vision errors, we will report that encoder re-architecture is unnecessary.
Why not just prompt, with chain-of-thought or visual marking? DCoT gains 2.47% on MM-Bench and reaches 38.4% on MM-Vet from 36.1% without training. However, its authors note that if the first grounding turn fails, later turns cannot recover. MATH-Vision also found chain-of-thought sometimes lowers accuracy. Prompting operates on top of whatever the encoder provides, so it is complementary rather than competing. We will apply a fixed chain-of-thought prompt to every arm to check that our gains survive it.
Why not just widen the encoder or raise resolution? Depth-versus-width results come from speech and text: Very Deep Self-Attention Networks for End-to-End Speech Recognition and SpanNorm. They are suggestive but not evidence for images. For that reason the 24-layer wide arm is part of the depth sweep, not an assumption. Resolution is held fixed across arms, and in a final check the best arm and the wide baseline are both rerun at doubled resolution. If doubling the wide baseline's resolution recovers the gap, the deep encoder's advantage was really about token detail, and the claim fails.
Why not just convert charts to tables or code? ChartQA pipelines that combine vision with table extraction reach 45.52% overall and about 30% on human-written questions. They struggle with color and layout references. Synthesize Step-by-Step requires a reliable extraction step that does not exist for unlabeled geometry. We report a table-extraction pipeline on ChartQA only, as a reference point.
Proposed Experiments
Perception is tested with the Chart101 protocol from Plot Twist, using its 5% range metric and collective per-image accuracy. We use its stress conditions: answer-bearing, removed and misleading titles, and added grid lines. As reference values, Plot Twist reports up to 79% with cue titles, the 30% range without them, and a drop for PaLI-3 from 29.5% to 21.3% with grid lines. Our robustness measure is the accuracy drop from cue-title to removed-title charts. A model that truly reads pixels should shrink that drop. ChartQA-H and ChartQA-M test whether elementary gains carry into compositional questions.
Reasoning is tested on MATH-Vision overall and per discipline. As reference points, the paper reports GPT-4o at 30.39% overall, about 18% on transformation geometry, near 50% on arithmetic, and chance at 5.86%. We expect encoder depth to help most on geometry-heavy disciplines. We will annotate gold textual figure descriptions for a stratified subset of about 400 problems. On this subset each arm is run with and without the description. An error that the description fixes counts as perception-attributed, and one that persists counts as reasoning-attributed. This gives the crossover test without relying on manual error taxonomies. MathVista and DocVQA check generality. MATH-500 measures how much text ability each arm loses after multimodal tuning, comparing against DPA's reported 36.4 versus 54.2.
Three outcomes would decide the question. First, does the final-minus-peak spatial-probe gap shrink with SpanNorm wiring and grow with PreNorm as depth increases? Second, does the deep SpanNorm encoder beat multi-layer readout on title-removed Chart101 accuracy? Third, does the error decomposition show encoder depth reducing perception-attributed errors and the DPA perceiver reducing reasoning-attributed ones? A yes to all three would establish that the depth budget should be split by error type. A no on the second would redirect the field toward cheap readout schemes.
References
- Perception Encoder: The best visual embeddings are not at the output of the network
- SpanNorm: Reconciling Training Stability and Performance in Deep Transformers
- Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset
- GeoSQA: A Benchmark for Scenario-based Question Answering in the Geography Domain at High School Level
- Deep Pre-Alignment for VLMs
- DCoT: Dual Chain-of-Thought Prompting for Large Multimodal Models
- Very Deep Self-Attention Networks for End-to-End Speech Recognition
- ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning
- Synthesize Step-by-Step: Tools, Templates and LLMs as Data Generators for Reasoning-Based Chart VQA
- Plot Twist: Multimodal Models Don't Comprehend Simple Chart Details