TL;DR
SynthDocBench is a fully synthetic benchmark designed to evaluate Vision Language Models (VLMs) on long-context visual document understanding. By systematically isolating variables like document length, layout structure, and chart types, it uncovers critical VLM failure modes—such as severe positional bias and a breakdown in chart comprehension over long contexts—that standard benchmarks mask.
What Problem Does This Paper Address?
While frontier VLMs are nearly saturating single-page document benchmarks (scoring >95% on DocVQA), real-world documents are long, visually dense, and structurally complex. When a model fails to answer a question about a 50-page corporate report containing text, tables, and charts, it is incredibly difficult to diagnose why. Is it failing because of the document's length? The chaotic layout? The specific chart type? Or the multi-step reasoning required?
Existing benchmarks rely on real-world documents where these variables are hopelessly confounded. To eliminate this diagnostic blind spot, the authors introduce SynthDocBench, a completely controlled, programmatically generated testing ground that varies document length, page depth, modality composition, and question difficulty as independent axes.
Methods
The authors developed a fully automated, LLM-based document generation pipeline that operates via a dual-layer design. Every chart in a document is simultaneously rendered as a visual D3.js graphic and saved as a hidden structured metadata block. This metadata allows the pipeline to programmatically derive deterministic, flawless ground-truth answers, completely bypassing the annotation bottlenecks and errors common in human-labeled datasets.
To ensure models cannot exploit shortcuts or spurious correlations (e.g., associating a specific topic with a specific visual style), the pipeline applies six distinct layout archetypes—such as Academic, Infographic, Dashboard, and Brutalist—with a 40% random layout override.
The query generation stage then parses these documents to construct three distinct question families across five difficulty levels (L1 to L5): chart-reading, cross-modal grounding, and complex multi-hop reasoning.
Datasets
The paper introduces SynthDocBench, which comprises 200 synthetic reports and 1,788 questions. The dataset is highly dense, averaging 51.1 pages, 16.7 charts, and roughly 20,568 words per document.
As shown in the benchmark landscape below, SynthDocBench occupies a unique frontier, offering significantly higher textual density and longer multi-page contexts than traditional chart or document VQA datasets like ChartQA, DocVQA, or MP-DocVQA.
Key Insights and Ideas
- The "Lost in the Middle" Effect Applies to Vision: Just like text-only LLMs, VLMs exhibit severe positional bias when processing long documents. For five out of the eight models tested, information located in the middle third of a document was systematically the hardest to retrieve and process, revealing a distinct U-shaped performance curve.
- Long-Context Chart Blindness: Models that excel at reading isolated charts experience a severe performance collapse when those same charts are buried inside a multi-page document.
- Perception, Not Reasoning, is the Primary Bottleneck: A deep-dive error analysis revealed that hard failures are dominated by "visual hallucinations"—where a model confidently reads an entirely incorrect value or scale from a dense chart or axis label—rather than a failure in downstream logical reasoning.
Key Results
The authors evaluated seven frontier VLMs using GPT-5 as a factual accuracy judge:
- Gemini-3.1-Pro led the pack by a wide margin, achieving an overall accuracy (ACC) of 72.5%.
- Qwen3.5-VL-122B (an open-weight MoE model) secured second place at 65.5%, outperforming several proprietary giants.
- Surprisingly, commercial models like GPT-4o (38.6%) and Claude-Sonnet-4.5 (31.4%) lagged significantly behind open-weight alternatives like Qwen3-VL-235B (58.6%).
- As difficulty scaled from L1 (direct lookup) to L5 (complex visual interpretation), performance degraded monotonically for almost all models. Gemini-3.1-Pro was the sole exception, maintaining a remarkably flat performance profile across difficulty tiers.
Limitations and Future Work
The authors note a potential "rendering-familiarity" confounder: SynthDocBench relies entirely on web-rendered HTML/D3.js charts. Because Gemini's training mixture likely includes a massive volume of web-rendered content, its substantial lead might partially stem from stylistic familiarity rather than superior long-context reasoning alone.
To address this, future work will explore alternative rendering backends and extend the benchmark to include even more diverse, chaotic document elements such as multi-page forms, complex financial tables, and mixed-layout corporate reports.
Summary updated