Document parsers now score above 90 on the most popular public benchmark. That benchmark only checks whether a model copies text, tables and formulas off the page correctly. It never checks whether the model can then use what it read: find a number in a table, apply a formula stated two pages earlier, respect a condition hidden in a footnote, and combine the results. We propose to generate documents whose content is a known reasoning problem. A random dependency graph of facts, of the kind used to test step-by-step reasoning in text, gets scattered across the body text, tables, formulas, footnotes and captions of a rendered page. Because the graph, the page markup and the answer are all known by construction, no human annotation is needed. A fresh, never-published instance can be drawn for every evaluation. Any wrong answer can also be traced to its cause: misreading the page, or failing to reason over correctly read content.
The payoff is a measurement that no current benchmark in this area provides. We can say how much of a model's document-question failure is perception and how much is reasoning. We can say how that split changes as the chain of dependencies gets longer. We can say whether where a fact sits on the page (table cell versus footnote) matters beyond whether the fact was transcribed correctly. This speaks to three live problems at once. Parsing scores may overstate document understanding. Static benchmarks leak into training data. Labeling realistic document reasoning data by hand is expensive and error-prone.
Core Claim
Our central hypothesis is that the errors of current document models are not simply the product of per-fact reading errors and text-only reasoning errors. Laying the same reasoning problem out across a page adds its own cost, and that cost grows with the number of sequential dependencies. If this is true, parsing leaderboards and text reasoning benchmarks will both mis-rank models on real document tasks. A procedurally generated, source-traceable benchmark can expose this without any manual labels.
Stated precisely, let $A_{\text{oracle}}(d)$ be a model's accuracy on depth-$d$ questions when given the true markup of the page. Let $p_i$ be its measured accuracy at extracting the individual premise $i$ from the page image. If reading and reasoning fail independently, end-to-end accuracy on a question using premises $i \in S$ should be
$$ \hat{A}_{\text{img}}(d) = A_{\text{oracle}}(d)\prod_{i \in S} p_i . $$
The product form comes from requiring every premise to be read correctly and the reasoning to succeed, with no interaction between the two. We define the layout interaction gap as $\hat{A}_{\text{img}}(d) - A_{\text{img}}(d)$, where $A_{\text{img}}(d)$ is observed end-to-end accuracy. A gap that is positive and grows with $d$ is the signature we are looking for. It would mean that holding facts in scattered page locations while reasoning is harder than reasoning over the same facts in clean text, even when each fact is readable in isolation.
Where This Sits Relative to Prior Work
Two lines of work each supply half of the method, and neither covers the other half.
Dynamic reasoning benchmarks. DyVal generates reasoning questions from random directed acyclic graphs (networks of facts where each node depends on earlier ones, with no loops). It controls depth and width, and shows three things: depth hurts far more than width, reversing the order of descriptions costs GPT-4 13.6% on reachability, and models strong on static math benchmarks score near zero on fresh instances. A²RBench adds formal cycle-consistency checks and shows a 17.7% drop when familiar symbols are renamed. Both motivate our design, and we reuse DyVal's graph generator and A²RBench's symbol remapping directly. Both are purely textual or grid-based, so they cannot see anything that happens between the pixels and the reasoning.
Source-traceable document benchmarks. PureDocBench renders pages from known HTML/CSS so the ground truth is exact. It found 2,580 annotation errors (12.08%) among 21,353 scored blocks of OmniDocBench. It also found that the best model reaches only about 74% average across its clean, digitally degraded and real-degraded tracks. We reuse its rendering-from-source approach and its degradation tracks. PureDocBench measures only transcription fidelity, not whether the transcribed content supports correct inference, and that is exactly the part we add.
Parsing as reasoning. Reading or Reasoning? reframes OCR as a structural reasoning problem, noting that output uncertainty on tables and formulas is an order of magnitude higher than on plain text. It trains with format-specific rewards to 90.41 on OmniDocBench. OvisOCR reaches 93.19 on OmniDocBench v1.5 with a 1B model. Both use "reasoning" to mean producing well-formed structure, and both are scored only on the benchmark PureDocBench found noisy and exposed. They are the systems whose document understanding we want to stress-test, not competitors to the benchmark. The closest paper to our question is STEM-POM. It shows GPT-4o reaches only 68.5% at identifying the role of math symbols from document context, which is evidence that reading a symbol is not the same as understanding it. It relies on 2,109 hand-labeled instances and tests single-symbol tagging rather than multi-step inference. Our generator removes the hand labeling and extends the question to chains of dependencies.
Research Program
The generator. We sample a dependency graph from DyVal's families: arithmetic, boolean and deductive logic, and reachability. We control depth $d$ from 1 to 6 and width $w$. Each node carries a premise, for example "the calibration offset $\kappa$ equals the value in row Q3 of Table 2 minus 4". The generator assigns each premise an element type: body sentence, table cell, displayed formula, inline formula, footnote, figure caption or sidebar. It also assigns a placement (page, column, distance in reading order from the premises it depends on). A templating layer wraps the premises in plausible domain prose drawn from several of PureDocBench's document styles: academic paper, financial report, textbook, exam. It adds distractor numbers and near-duplicate table rows, and renames symbols A²RBench-style so that memorized conventions cannot help. Pages are compiled from HTML/CSS and LaTeX, then passed through PureDocBench-style digital degradation, plus a real-degradation track made by printing and scanning a subset.
Each instance ships four aligned artifacts: the page images, the true markup, the flat-text version of the graph, and the answer. No human labels anything. A small human audit of a few hundred instances checks only that each question is answerable and each answer correct. We target an audit error rate well under 1%, against the 12.08% found in hand-annotated OmniDocBench.
The decomposition. For each model we measure accuracy under four input conditions on the same graphs. The first is flat text, which reproduces DyVal. The second is oracle markup, the true structured page. The third is parsed markup, meaning the output of a document parser, either the model's own or the strongest available specialist. The fourth is the raw image, end to end. We also ask probe questions that request a single premise ("what is the value in row Q3 of Table 2?"), which gives the per-premise reading accuracies $p_i$ in the equation above. The gaps between conditions attribute failure to perception (parsed versus oracle), to document structure (oracle versus flat text), and to the interaction term defined above. Because element type and placement are randomized independently of graph content, we can estimate the cost of a premise being in a footnote or table at matched reading accuracy.
Contamination resistance as a tested property, not a slogan. Every evaluation uses freshly seeded instances. We will verify that a model's score is stable across seeds. We will also run a deliberate leak experiment: fine-tune a small open model on one fixed seed set and show that its score on that set inflates while its score on fresh seeds does not.
Baselines
The naive reference point is the flat-text condition, which is simply DyVal on the same graphs. It asks whether a document adds anything beyond the reasoning problem itself. If image and oracle-markup accuracy match flat-text accuracy once reading errors are accounted for, the benchmark collapses into "DyVal plus OCR" and our core claim is false. We run it with identical prompts, the same answer extraction and the same decoding settings for every model.
The strongest existing system is a two-stage pipeline: the best available document parser feeds its markdown to a strong text reasoner. On transcription, the strongest reported parsing number in our file is OvisOCR's 93.19 Overall on OmniDocBench v1.5 (OvisOCR). The format-decoupled RL model reports 90.41 on OmniDocBench (Reading or Reasoning?). On the harder source-traceable setting, the best reported average is about 74% (PureDocBench). For reasoning over parsed document content we found no reported result on Lacuna for any system. The benchmark will therefore establish the first numbers rather than compare against a published one.
We will run OvisOCR and MinerU where weights are available, plus Marker. Each pipeline is paired with the same text reasoner, so that the pipeline differs from the oracle condition only in parse quality. The strongest end-to-end competitors are frontier general vision-language models (models that take page images and text together): GPT-4o, Gemini 2.5 Pro and Qwen3-VL-235B. Open models such as Qwen2-VL are included for reproducibility. All get the same image resolution and the same token budget.
We add a prompting baseline that splits the question into sub-questions, following the divide-and-conquer analysis in When Splitting Makes Stronger. It gets the same total generation budget as plain chain-of-thought.
Two ablations isolate the ingredient we claim matters, which is layout placement. The first renders the flat-text graph as a plain image with no tables, footnotes or columns. The second holds element types fixed but shuffles placement so that dependent premises are adjacent in reading order.
For a benchmark, "beating" the existing references means predicting something they cannot. Our success criterion has two parts. First, model rankings on image-condition accuracy at depth 4 and above should correlate only weakly with OmniDocBench and PureDocBench parsing rankings; we set a Spearman correlation below 0.5 as the bar. Second, the layout interaction gap should be significantly positive and increase with depth for at least the majority of models, with confidence intervals from bootstrapping over fresh seeds.
Why not just…?
Why not just scale up a frontier vision-language model and assume reading and reasoning come together? Scale does not straightforwardly help on the reading side. OvisOCR's 1B model beats models up to 200× larger on OmniDocBench v1.5. PureDocBench finds specialists at or below 4B rivaling models 5–100× larger on clean pages. GPT-4o manages only 68.5% on STEM-POM even with full manuscript context. Frontier models are more robust to degradation, however: PureDocBench reports they lose 0.99 and 8.52 points versus 4.90 and 14.21 for pipelines. So they may well win the end-to-end condition. We do not argue against this. The benchmark measures it, and the decomposition says whether any win comes from better reading or from better reasoning over scattered facts.
Why not just parse with the best specialist and hand the markdown to the best text reasoner? This is the most serious practical threat, and it is a baseline rather than a straw man. If the pipeline's accuracy equals the oracle-markup accuracy times its reading accuracy, the independence model holds. In that case the right conclusion is that document understanding reduces to parsing plus text reasoning, and the parsing leaderboards are adequate proxies. Our core claim would be refuted. The parsed-versus-oracle comparison settles this directly.
Why not just render DyVal prompts as images? That tests OCR of plain paragraphs, and it is exactly our first layout ablation. If adding tables, footnotes and cross-page placement does not lower accuracy beyond the plain-text render, then layout adds nothing and the extra generator machinery is unjustified. We expect the opposite, because DyVal already shows that description order alone moves GPT-4 by 13.6%, and page layout scrambles order far more aggressively. That remains an expectation to test, not a result.
Why not just collect fresh real documents and write questions for them, LiveBench-style? Freshness would address contamination, as LiveBench does for text. Every question would still need expert annotation. PureDocBench's audit shows hand annotation of documents carries double-digit error rates, and STEM-POM needed a dedicated labeling tool for about two thousand items. Real documents also give no control over depth or placement, so the decomposition would be impossible. Realism is the genuine weakness of synthetic pages. We address it by checking that parser rankings on our renders agree with PureDocBench's real-degraded track, and by including the print-and-scan track.
Why not just use chain-of-thought prompting to fix the reasoning half? Prior evidence suggests intermediate steps are often unsound even when answers are right (Evaluating Intermediate Reasoning of Code-Assisted LLMs). External plans also do not stop degradation as complexity grows (Beyond Code). We include both chain-of-thought and decomposed prompting as baselines. If either closes the layout interaction gap, that is a useful finding in its own right, and it would point to a cheap fix rather than a new training objective.
Proposed Experiments
The main study evaluates roughly a dozen models across depths 1–6, all element types, and the clean, digitally degraded and real-degraded tracks. It uses several hundred fresh instances per cell, matching DyVal's 500 per dataset where budget allows, and reports all four input conditions plus probe accuracies.
The first decisive check is the fit of the independence prediction. We will plot predicted against observed image accuracy by depth and test whether the gap grows with $d$. A flat, near-zero gap refutes the core claim.
The second check is the element-type effect at matched reading accuracy. We will test whether a premise in a footnote or table lowers question accuracy more than the same premise in body text, even when its probe accuracy is equal. This is the document analogue of DyVal's order effect and A²RBench's remapping drop.
The third check is ranking divergence: rank correlation between our end-to-end scores and published OmniDocBench and PureDocBench orderings, for the models where both exist.
The fourth check is contamination resistance. We will measure seed-to-seed stability, and run the leak experiment in which a model fine-tuned on a fixed seed set gains on that set but not on fresh seeds.
The fifth check is label quality: the human audit rate on a random sample, reported against the 12.08% OmniDocBench figure.
As a secondary question, following DyVal's finding that training on its graphs improved static reasoning benchmarks, we will fine-tune a small open document model on generated instances. We then test whether it transfers to PureDocBench transcription and to STEM-POM symbol tagging. This addresses whether the generator is also a label-free training source, but the benchmark and the decomposition stand without it.
References
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- A²RBench: An Automatic Paradigm for Formally Verifiable Abstract Reasoning Benchmark Generation
- How Far Is Document Parsing from Solved? PureDocBench: A Source-TraceableBenchmark across Clean, Degraded, and Real-World Settings
- OmniDocBench: Benchmarking Diverse PDF Document Parsing with Comprehensive Annotations
- Reading or Reasoning? Format Decoupled Reinforcement Learning for Document OCR
- OvisOCR: End-to-End Document Parsing via Aligning Specialized Perception with General Reasoning
- STEM-POM: Evaluating Language Models Math-Symbol Reasoning in Document Parsing
- When Splitting Makes Stronger: A Theoretical and Empirical Analysis of Divide-and-Conquer Prompting in LLMs
- LiveBench: A Challenging, Contamination-Limited LLM Benchmark
- Evaluating Intermediate Reasoning of Code-Assisted Large Language Models for Mathematics
- Beyond Code: Evaluate Thought Steps for Complex Code Generation