Language models reason worse when forced to answer in a rigid format such as JSON. On a symbolic task, GPT-3.5 falls from about 56% to 25% when the schema puts the answer before the reasoning (Let Me Speak Freely?). On MATH500, a 14B model drops from 76% to 47% under grammar-constrained decoding, meaning decoding that masks out any token that would break the required syntax (Draft-Conditioned Constrained Decoding). Every one of these measurements comes from text-only models. Vision-language models are increasingly deployed exactly where structured output is mandatory: pulling fields from invoices, turning charts into tables, emitting tool calls about what a camera sees. So it matters whether the same tax appears there, and what it is made of. A vision-language model can fail under a format for two different reasons. It may still read the image correctly and then reason badly. Or it may stop looking at the image. Those two failures call for different fixes.
This note proposes to split the multimodal format tax into a perception part and a reasoning part. The split uses benchmarks where the image's content is also available as text: charts with their underlying data tables. The study then asks which part each available remedy actually repairs. The remedies are reordering the schema, letting the model think freely and then constraining only the final answer, giving formatted few-shot examples, and a small change to the second, instruction-tuning stage of training. The prediction is that inference-time remedies repair the reasoning part but leave a perception part. That residual should be largest when the formatted output is itself a visual extraction, such as a table read off a chart. It should be traceable to how two-stage models are trained, and a cheap change to stage-two data should remove it.
Core Claim
Most open vision-language models are trained in two stages. In stage one, a small projection layer learns to turn image-encoder features into "soft tokens" the frozen language model can read (An Empirical Study of Scaling Instruct-Tuned Large Multimodal Models). In stage two, the whole system is instruction-tuned. LLaVA-OneVision extends this to three stages. In none of these recipes does the training data deliberately pair schema-constrained outputs with answers that require looking at the image.
Our hypothesis is specific. Under a format constraint, these models shift attention away from visual tokens and toward syntax-tracking. As a result, part of the multimodal tax survives even the best text-side remedy, which is a free-form draft followed by constrained projection. Two pieces of prior work point the same way, and they motivate the hypothesis. Reading or Reasoning? finds that document-OCR models have roughly an order of magnitude higher output entropy on formatted spans (formulas, tables) than on plain text. Multimodal-CoT shows that grounding rationales in visual features is what removes most hallucinations. If formatting weakens that grounding, the multimodal tax is not simply the text tax plus images.
To make the split concrete, let $A_{\text{free}}$ and $A_{\text{fmt}}$ be accuracy with the image under free-form chain-of-thought and under a given format condition. Chain-of-thought means writing out intermediate reasoning before answering. Let $T_{\text{free}}$ and $T_{\text{fmt}}$ be the same accuracies when the image is replaced by its gold data table rendered as text. We define
$$\text{reasoning tax} = T_{\text{free}} - T_{\text{fmt}}, \qquad \text{perception tax} = (A_{\text{free}} - A_{\text{fmt}}) - (T_{\text{free}} - T_{\text{fmt}}).$$
The reasoning tax is the loss that appears even when perception is handed to the model. The perception tax is the extra loss that appears only when the model must read the image itself. The decomposition is an operational estimate, not an identity: reading a table as text differs from reading a chart. For that reason we confirm it with two direct measurements. The first is the share of attention that answer-relevant tokens place on image tokens. The second is the gap between accuracy with the image and accuracy with the image blanked out.
Research Program
The first part is measurement. We take two open two-stage models: LLaVA-OneVision, whose staged recipe is documented, and Qwen2.5-VL-3B, whose induction heads are already mapped by Dissecting Multimodal In-Context Learning. Induction heads are attention heads that copy patterns forward from earlier in the context. Each model is evaluated under six conditions. The first is free-form chain-of-thought. The second is a JSON instruction with the answer field first, and the third puts the reasoning field first; these are the two orderings studied by Let Me Speak Freely?. The fourth is full grammar-constrained decoding. The fifth is think-then-constrain, the Draft-Conditioned Constrained Decoding (DCCD) idea run inside one sequence with the visual key-value cache reused, so that nothing is re-encoded. The sixth is the reasoning-first JSON instruction plus three to five formatted few-shot demonstrations.
We use two kinds of target output. A scalar output asks for one numeric or categorical answer wrapped in JSON. A structured-extraction output requires the formatted content itself to be read from the image, for example a JSON table of a chart's series.
The second part is mechanism. Few-shot demonstrations are known to compress into a "task vector": an internal summary at the last demonstration token that steers the rest of generation (In-Context Learning Creates Task Vectors). We extract task vectors from formatted and from free-form demonstrations and patch each into a zero-shot pass. This tests whether the formatted vector by itself lowers image-attention share and raises the blind-image gap. If it does, formatted examples pass the perception tax along through the same steering channel that makes in-context learning work. This would extend the task-vector analysis beyond the single-token tasks it was originally tested on. We also knock out the induction heads identified in the circuits paper to see whether format-matching from demonstrations competes with visual copying for the same heads.
The third part is a training intervention aimed at the perception tax only. During stage two we add a small LoRA mixture, LoRA being low-rank adapters that update only a few added parameters. The mixture contains grounded formatted rationales. We build it by self-distillation: the model's own correct free-form chart and document answers are rewritten into the target schema, and a reasoning field quotes the values read from the image. The point is to give the model the one pairing its curriculum lacks, structure together with grounding, so that it can answer in a single constrained pass. This differs from FD-RL (Reading or Reasoning?), which uses format-specific rewards to sharpen transcription fidelity with no deliberation step. Our goal is reasoning over images under a schema, and we diagnose before we treat.
Baselines
The naive reference point is free-form chain-of-thought with no format, parsed by regular expression. It is the ceiling every format condition is measured against, and it defines $A_{\text{free}}$ and $T_{\text{free}}$. The second baseline is plain JSON with reasoning first and nothing else. This is what a practitioner does by default, and it isolates the contribution of schema reordering.
The strongest existing method is DCCD from Draft-Conditioned Constrained Decoding. On text, it raises 14B MATH500 accuracy from 47% under standard constrained decoding to 58.6%, against 76% unconstrained. It raises 1B strict structured accuracy on GSM8K from 15.2% to 39.0%. We run it two ways. One is faithful to the paper with two passes. The other is a single sequence with a shared visual cache. The single-sequence version removes the latency argument that would otherwise favour our training intervention unfairly. We found no reported result on Lacuna for DCCD, or for any format-tax measurement, on a vision-language model or on ChartQA, DocVQA, or ScienceQA, so its multimodal numbers are produced here.
The training intervention has a data-matched control: an equal number of stage-two LoRA steps on the same images with free-form rationales. This separates "grounded formatted data" from "more data on these images." All model conditions use the same checkpoints, the same images, the same prompt-tuning budget (a fixed number of schema and prompt variants, with the best chosen on a held-out split for every method), and the same decoding temperature.
The intervention counts as beating DCCD only under specific conditions. On structured-extraction tasks, it must exceed single-sequence DCCD accuracy by more than the seed-to-seed spread. On scalar tasks, it must match DCCD within that spread while generating fewer tokens. If it only beats the free-form-data control, the result shows that data composition matters but does not justify the method over inference-time decoupling.
Why not just…?
Why not just put the reasoning field before the answer field? It is cheap and it fixes the ordering failure shown in Let Me Speak Freely?. The constrained-decoding results in Draft-Conditioned Constrained Decoding show that token masking costs accuracy even when deliberation is allowed. The reasoning-first JSON condition measures directly how much of the multimodal tax reordering removes. Our hypothesis predicts it removes little of the perception part.
Why not just think freely and then constrain only the answer, in one sequence? This is the real threat. With the visual cache reused it costs little extra memory, which weakens the latency objection raised against two-pass pipelines. If single-sequence DCCD drives the perception tax to zero on structured-extraction tasks, the training intervention is unnecessary. In that case we report the decomposition and the benchmark results as the contribution. Our reason to expect a residual is the entropy finding in Reading or Reasoning?. When the formatted span is itself the extraction, the projection step re-reads the image under the format, and a correct free-form draft does not guarantee a correct table.
Why not just use a bigger model? The text-only tax persists at 14B (76% to 47% on MATH500), and DCCD reports that pairs of small decoupled models beat larger constrained ones. We run both LLaVA-OneVision sizes available to us. If the perception tax shrinks with scale while the reasoning tax does not, that finding is itself informative about where capacity goes.
Why not just show formatted few-shot examples? Format-restricting instructions already underperform natural-language reasoning in text (Let Me Speak Freely?). The classification paradox points the other way: on DDXPlus, JSON mode raised Gemini-1.5-Flash from 41.6% to 60.3%. Our few-shot condition and the task-vector patching measure the trade-off directly. If formatted demonstrations help scalar classification-style questions but raise the blind-image gap on extraction, that confirms the paradox carries over to images and locates its cost in perception.
Proposed Experiments
The main testbed is ChartQA (ChartQA). It suits this study because charts come with underlying data tables, which makes the table-substitution condition possible, and because it mixes extraction with arithmetic. Its originally reported best model reached 45.52%. That number is far below current models, so it serves only as historical context.
DocVQA (DocVQA), where LLaVA-OneVision-72B reports 91.3% free-form, supplies key-value extraction into JSON. Gold text substitution there uses the dataset's OCR transcripts, which makes it an approximation. ScienceQA (Multimodal-CoT) supplies multiple-choice scientific reasoning, where we expect a classification-paradox benefit. GSM8K is run on the language backbones alone, to check that the text-only tax replicates in our setup. Metrics are exact-match accuracy (or ANLS on DocVQA), schema validity, image-attention share over answer tokens, the blind-image gap, and generated tokens per answer. Results are averaged over three seeds where sampling is involved.
The decisive check is a sign-and-size test on the residual perception tax after single-sequence DCCD, split by output type. Our hypothesis predicts three things: the residual is near zero for scalar outputs, clearly positive for structured extraction, and accompanied by lower image-attention share. It also predicts that the grounded-format LoRA removes this residual where the data-matched free-form control does not. Each of these can fail independently. A zero residual on extraction refutes the perception-tax story. Unchanged attention share refutes the mechanism. A control that does as well as the intervention refutes the claim about curriculum design.
References
- Let Me Speak Freely? A Study On The Impact Of Format Restrictions On Large Language Model Performance
- The Hidden Cost of Structured Generation in LLMs: Draft-Conditioned Constrained Decoding
- An Empirical Study of Scaling Instruct-Tuned Large Multimodal Models
- LLaVA-OneVision: Easy Visual Task Transfer
- Reading or Reasoning? Format Decoupled Reinforcement Learning for Document OCR
- Multimodal Chain-of-Thought Reasoning in Language Models
- Dissecting Multimodal In-Context Learning: Modality Asymmetries and Circuit Dynamics in modern Transformers
- In-Context Learning Creates Task Vectors
- ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning
- DocVQA: A Dataset for VQA on Document Images