Multimodal models now score above 95% on document question-answering benchmarks such as DocVQA. Yet they fall below 50% when the answer has to be computed from the text in an image rather than copied out of it. When such a model gets a question about an invoice or a chart wrong, a static benchmark cannot say why. The model may have misread a number. It may have read every number correctly and then reasoned badly. Or its correct answers on other items may have come from remembering leaked test items rather than working them out.
This proposal builds a generator of synthetic text-rich documents in which every item comes with its hidden data table and an executable reasoning program. That lets us score each answer three ways: did the model read the fields it needed, did it reason correctly given those fields, and does its accuracy survive when the same data is redrawn so that memorization cannot help. With that decomposition, we test one specific question that existing work leaves unresolved: when step-by-step reasoning (chain-of-thought) helps on document questions, does it also quietly corrupt the values the model reads off the page?
Core Claim
The headline numbers on text-rich reasoning benchmarks mix three error sources that call for different fixes: reading failures, reasoning failures given correct reading, and recall of leaked items. We claim these can be pulled apart item by item, using documents rendered from known latent data, and that doing so will settle two open disagreements.
The first disagreement concerns chain-of-thought. It appears to help on document reasoning, raising InternVL2.5 by over 10% on Reasoning-OCR. It appears to hurt on simple perception, where most models show negative stability scores in MME-CoT. These findings come from different items, so we cannot tell whether the gain and the loss coexist inside the same answer. Our hypothesis is that they do. The written chain re-states values from the image, and some fraction of those re-stated values drift from what is actually printed, even when the model's direct reading was right. We call this transcription drift, and our design measures it directly because the true values are known.
The second disagreement concerns contamination. Redrawing a public chart in a new style makes accuracy fall, but Plot Twist shows that style changes alone, such as adding grid lines, hurt perception. A raw drop therefore cannot be blamed on memorization. Reading probes resolve this confound. If a model still reads every needed field correctly on the redrawn item but loses the final answer, the lost accuracy was not reasoning from the page.
Relation to Prior Work
Three benchmarks motivate the gap but leave the decomposition open. Reasoning-OCR hand-builds 150 multi-hop questions over 140 text-rich images and finds a "decision wall" where some models score 0%. OCR-Reasoning builds 1,069 items in which only 2.3% of answers appear verbatim in the OCR text, and no model exceeds 50%. It also shows that replacing the image with raw OCR text hurts, which tells us that layout carries information. MME-CoT scores reasoning chains against annotated key steps and documents the overthinking effect. All three are static and human-annotated. None knows the true value of every field a question depends on, so none can separate reading errors from reasoning errors within a single answer. We reuse their task categories as templates for our generator and their models as reference points.
Three lines of procedural evaluation supply the method but cover different ground.
- DynaMath writes 501 Python seed programs that render visual math variants. It reports worst-case accuracy over ten variants, finding 35.3% for Claude 3.5 Sonnet against 64.8% on average. We adopt its seed-program design and worst-case metric, but apply them to dense documents (multi-column reports, forms, timetables, tables) rather than function plots and geometry.
- DyVal generates text-only reasoning problems from directed acyclic graphs (networks of steps where each step depends only on earlier ones) with controlled depth. We use the same kind of graph as the hidden reasoning program behind each document question, which gives reasoning depth as a dial independent of visual difficulty.
- Vision-Language Bootstrapping perturbs natural images with diffusion inpainting. It cuts measured contamination of MME against LAION-100M from about 33% to about 10%. Inpainting cannot guarantee that the text and numbers in an edited document remain exactly right, whereas symbolic rendering can.
Two papers constrain the design. GeoSQA shows that models fail even when diagrams are fully transcribed by humans. This is why every item includes an oracle-text condition: if reasoning failures persist there, the bottleneck is not perception. Can Large Language Models Reason? A Characterization via 3-SAT shows that longer chains do not fix combinatorially hard problems. We therefore keep the reasoning programs to arithmetic, comparison, and conditional lookups, where chains are known to help, so that any harm from chain-of-thought cannot be blamed on intractable search.
On the OCR side, LayoutParser supplies the layout-detection pipeline we use as a baseline. The hybrid denoising and thresholding work of A Hybrid Approach to Detect and Identify Text in Picture tells us which physical degradations (blur, uneven backgrounds, edge defects) matter for recognition. We use those degradations as a second, independent dial on reading difficulty.
Research Program
The generator is a set of seed programs. Each seed defines three things:
- A document type. Examples are an invoice, a two-column financial report, a bus timetable, a lab-results form, and a chart paired with a table.
- A sampler for the hidden data. This draws a table of entities, numbers, dates, and units.
- A reasoning template. This is a small graph of operations over named fields, such as filter, compare, sum, and conditional selection. It follows the six reasoning categories of Reasoning-OCR and the numerical, enumerative, and spatial categories of OCR-Reasoning.
A renderer draws the data with sampled layout settings: column count, font, whether a field appears in a table cell or in running prose, and distractor fields. A separate degradation stage applies skew, blur, uneven illumination, and binarization artifacts at controlled strength. Answers are computed by executing the graph, so correctness never depends on a judge model. That avoids the LLM-as-judge bias OCR-Reasoning acknowledges.
Every item ships with four things:
- The final question.
- Reading probes. These are one-step questions such as "what is the unit price on line 3?", one for every field the reasoning graph touches.
- An oracle-text rendering. This is a markdown version of the document that preserves layout.
- An OCR-pipeline transcription. This is produced by running LayoutParser block detection followed by an open OCR engine.
For a given model and item, the reading score is collective. It counts as correct only if every needed probe is answered correctly, following the collective-accuracy idea in Plot Twist. Reasoning accuracy is measured on the subset of items where reading is collectively correct, and separately in the oracle-text condition.
Transcription drift is defined for chain-of-thought outputs. We parse every field value the chain states and compare it with ground truth. We restrict this to fields the model read correctly when asked directly, which isolates errors introduced by the chain itself:
$$\text{drift} = \frac{\#\{\text{needed fields stated wrongly in the chain but read correctly by direct probe}\}}{\#\{\text{needed fields stated in the chain and read correctly by direct probe}\}}$$
Here the numerator counts needed fields that the model got right as a direct probe answer but misquoted inside its own reasoning, and the denominator counts all needed fields that were both stated in the chain and read correctly by probe. A drift rate clearly above zero is direct evidence for the mechanism MME-CoT infers only indirectly.
The contamination arm redraws public items. We take ChartQA items whose underlying data can be reconstructed and draw each one twice. One version keeps the data and changes the style. The other keeps the style and changes the numbers, with answers recomputed. We then compare accuracy on original and redrawn versions, conditioned on reading probes being correct. A model reasoning from the page should be indifferent to the redraw once reading is controlled. A model recalling memorized answers should lose accuracy exactly on the items where it still reads correctly.
Baselines
Direct answering. The naive reference point asks each model for the answer with no reasoning, on identical generated items at matched resolution and decoding settings. Every other condition is measured as a change from this one.
Chain-of-thought prompting. This is the most widely reported improvement on this exact question, and the condition whose net effect we want to decompose. The cited figure is the over-10% gain for InternVL2.5 on Reasoning-OCR. We run it with the same models and the same token cap across items, and include deliberative "thinking" models such as GLM-4.1V-Thinking and QVQ-72B in their native mode.
Strongest reported systems. For absolute calibration, the strongest numbers in the file are GPT-4o at 68.1% accuracy on Reasoning-OCR, and DouBao-1.5-Vision-Pro at 46.8% with OpenAI-o1 at 44.4% on OCR-Reasoning. We will run the open-weight subset of those models on both static benchmarks and on our generated items. If our items are valid, model rankings on them should agree with rankings on the static sets. No reported result was found on Lacuna for any model on procedurally generated text-rich documents, so the generated-item numbers will be new.
OCR pipeline followed by a text-only language model. The LayoutParser-plus-OCR transcription is passed to a strong text model with chain-of-thought. This is the classical answer to the reasoning gap and the natural competitor to an end-to-end multimodal model.
Oracle text. The text model receives the perfect transcription instead. This is an upper bound on what a reading fix alone could achieve.
Visual-guidance prompting. DCoT draws boxes around salient regions and retrieves example chains, gaining 2.47% on MM-Bench. We run it on the same items to see whether visual guidance reduces drift.
Read-then-reason ablation. This isolates our key ingredient. The model first writes out the needed fields as a structured table without reasoning, then reasons over its own table, optionally with the image hidden. If drift is the mechanism by which chain-of-thought hurts, this ablation should keep chain-of-thought's reasoning gains while recovering the lost reading accuracy.
What counts as beating a baseline. The read-then-reason condition beats plain chain-of-thought only if it raises final accuracy on deep reasoning graphs, measured on held-out seeds with confidence intervals from item-level bootstrap, without lowering collective reading accuracy.
Why not just…?
Why not just regenerate documents DynaMath-style and report worst-case accuracy? That would yield a contamination-resistant leaderboard. It would still conflate reading and reasoning, because a worst-case failure could come from either. DynaMath's own error analysis has to be done by hand: 33.1% figure reading and 26.6% reasoning for Claude 3.5 Sonnet. Our probes and oracle-text condition automate that split for every item. The worst-case metric remains one of our reported numbers, not the contribution.
Why not just run OCR and give the text to a strong language model? This is a real threat on clean, single-column documents, and the pipeline baseline exists to test it. OCR-Reasoning reports that stripping the image and keeping raw OCR text degrades reasoning, but that comparison used unstructured text. Our layout-preserving oracle text is a stronger version of this alternative. If oracle text closes nearly all of the gap across layouts and degradation levels, the right conclusion is that end-to-end models should transcribe first, and we will report that. The degradation dial shows where pipeline OCR errors reopen the gap.
Why not just apply an existing contamination detector to ChartQA or DocVQA? On The Fragility of Benchmark Contamination Detection in Reasoning Models reports that statistical detectors break under paraphrasing and re-rendering, which is exactly the setting of multimodal documents. Our test needs no access to training data. It asks only whether accuracy that survives correct reading also survives redrawing.
Why not just tell the model to transcribe before reasoning? That is our read-then-reason ablation, and it may be the most useful practical outcome. The decomposition is what explains whether, and on which documents, the prompt works. It also guards against a false positive: if the prompt helps only because it adds tokens, a length-matched chain-of-thought control will show it.
Proposed Experiments
We build about 60 seed programs across five document types and draw 20 variants per seed. Reasoning depth runs from one to four operations, and degradation from clean to heavily degraded. A further 15 seeds are kept private and released later to check that results hold on fresh templates.
Generator validity. Before any analysis, we check that model rankings on generated items correlate with rankings on Reasoning-OCR and OCR-Reasoning. A low correlation would mean our documents test something else. In that case we revise the templates rather than report results.
Effect of chain-of-thought. The main experiment crosses models with direct, chain-of-thought, and read-then-reason conditions. For each cell we report collective reading accuracy, reasoning accuracy given correct reading, oracle-text accuracy, and drift.
The decisive check for the chain-of-thought claim is this. On items where every needed field was read correctly by direct probe, we compare final accuracy under direct answering and under chain-of-thought, split by whether the chain quoted every needed value correctly. If chains that drift account for most of the cases where chain-of-thought turns a correct direct answer into a wrong one, the hypothesis is supported. If drift stays near zero while chain-of-thought still hurts perception items, then MME-CoT's effect lives somewhere other than value transcription, and the hypothesis is wrong.
Recall versus reasoning. The decisive check for the contamination claim compares original and redrawn ChartQA items, restricted to items whose reading probes are all correct in both versions. A significant drop on that restricted set measures accuracy that came from recall rather than from reading the page. This number cannot be obtained from raw accuracy drops, which Plot Twist shows are confounded with perceptual fragility.
Finally, we plot the oracle-text gap against degradation strength and layout complexity. This shows where better OCR preprocessing would pay off, and where reasoning, not reading, is the binding limit.
References
- Reasoning-OCR: Can Large Multimodal Models Solve Complex Logical Reasoning Problems from OCR Cues?
- MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency
- Plot Twist: Multimodal Models Don't Comprehend Simple Chart Details
- OCR-Reasoning Benchmark: Unveiling the True Capabilities of MLLMs in Complex Text-Rich Image Reasoning
- DynaMath: A Dynamic Visual Benchmark for Evaluating Mathematical Reasoning Robustness of Vision Language Models
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- Dynamic Multimodal Evaluation with Flexible Complexity by Vision-Language Bootstrapping
- GeoSQA: A Benchmark for Scenario-based Question Answering in the Geography Domain at High School Level
- Can Large Language Models Reason? A Characterization via 3-SAT
- LayoutParser: A Unified Toolkit for Deep Learning Based Document Image Analysis
- A Hybrid Approach to Detect and Identify Text in Picture
- DCoT: Dual Chain-of-Thought Prompting for Large Multimodal Models
- On The Fragility of Benchmark Contamination Detection in Reasoning Models