SynthDocBench: Controlled Benchmark for Long-Context Visual Document Understanding

Abhigya Verma, Khyati Mahajan, Amit Kumar Saha, Shruthan Radhakrishna, Sagar Davasam, Vikas Yadav, Sai Rajeswar Mudumba

TL;DR

SynthDocBench is a fully synthetic benchmark designed to evaluate Vision Language Models (VLMs) on long-context visual document understanding. By systematically isolating variables like document length, layout structure, and chart types, it uncovers critical VLM failure modes—such as severe positional bias and a breakdown in chart comprehension over long contexts—that standard benchmarks mask.

What Problem Does This Paper Address?

While frontier VLMs are nearly saturating single-page document benchmarks (scoring >95% on DocVQA), real-world documents are long, visually dense, and structurally complex. When a model fails to answer a question about a 50-page corporate report containing text, tables, and charts, it is incredibly difficult to diagnose why. Is it failing because of the document's length? The chaotic layout? The specific chart type? Or the multi-step reasoning required?

Existing benchmarks rely on real-world documents where these variables are hopelessly confounded. To eliminate this diagnostic blind spot, the authors introduce SynthDocBench, a completely controlled, programmatically generated testing ground that varies document length, page depth, modality composition, and question difficulty as independent axes.

Methods

The authors developed a fully automated, LLM-based document generation pipeline that operates via a dual-layer design. Every chart in a document is simultaneously rendered as a visual D3.js graphic and saved as a hidden structured metadata block. This metadata allows the pipeline to programmatically derive deterministic, flawless ground-truth answers, completely bypassing the annotation bottlenecks and errors common in human-labeled datasets.

To ensure models cannot exploit shortcuts or spurious correlations (e.g., associating a specific topic with a specific visual style), the pipeline applies six distinct layout archetypes—such as Academic, Infographic, Dashboard, and Brutalist—with a 40% random layout override.

Synthetic visual document generation pipeline

The query generation stage then parses these documents to construct three distinct question families across five difficulty levels (L1 to L5): chart-reading, cross-modal grounding, and complex multi-hop reasoning.

Query generation pipeline component

Datasets

The paper introduces SynthDocBench, which comprises 200 synthetic reports and 1,788 questions. The dataset is highly dense, averaging 51.1 pages, 16.7 charts, and roughly 20,568 words per document.

As shown in the benchmark landscape below, SynthDocBench occupies a unique frontier, offering significantly higher textual density and longer multi-page contexts than traditional chart or document VQA datasets like ChartQA, DocVQA, or MP-DocVQA.

Landscape of benchmarks comparing page count and text tokens

Key Insights and Ideas

  • The "Lost in the Middle" Effect Applies to Vision: Just like text-only LLMs, VLMs exhibit severe positional bias when processing long documents. For five out of the eight models tested, information located in the middle third of a document was systematically the hardest to retrieve and process, revealing a distinct U-shaped performance curve.
  • Long-Context Chart Blindness: Models that excel at reading isolated charts experience a severe performance collapse when those same charts are buried inside a multi-page document.
  • Perception, Not Reasoning, is the Primary Bottleneck: A deep-dive error analysis revealed that hard failures are dominated by "visual hallucinations"—where a model confidently reads an entirely incorrect value or scale from a dense chart or axis label—rather than a failure in downstream logical reasoning.

Key Results

The authors evaluated seven frontier VLMs using GPT-5 as a factual accuracy judge:

  • Gemini-3.1-Pro led the pack by a wide margin, achieving an overall accuracy (ACC) of 72.5%.
  • Qwen3.5-VL-122B (an open-weight MoE model) secured second place at 65.5%, outperforming several proprietary giants.
  • Surprisingly, commercial models like GPT-4o (38.6%) and Claude-Sonnet-4.5 (31.4%) lagged significantly behind open-weight alternatives like Qwen3-VL-235B (58.6%).
  • As difficulty scaled from L1 (direct lookup) to L5 (complex visual interpretation), performance degraded monotonically for almost all models. Gemini-3.1-Pro was the sole exception, maintaining a remarkably flat performance profile across difficulty tiers.

Limitations and Future Work

The authors note a potential "rendering-familiarity" confounder: SynthDocBench relies entirely on web-rendered HTML/D3.js charts. Because Gemini's training mixture likely includes a massive volume of web-rendered content, its substantial lead might partially stem from stylistic familiarity rather than superior long-context reasoning alone.

To address this, future work will explore alternative rendering backends and extend the benchmark to include even more diverse, chaotic document elements such as multi-page forms, complex financial tables, and mixed-layout corporate reports.

Summary updated

Abstract

Vision language models (VLMs) have achieved strong performance on visual document understanding benchmarks such as DocVQA, ChartQA, and MMLongBench-Doc. However, real-world documents combine multiple factors such as length, layout complexity, modality, and question difficulty, which makes it difficult to attribute model failures to specific causes. We introduce SynthDocBench, a fully synthetic benchmark for long-context visual document understanding that systematically controls factors including document length, layout structure, modality composition, and question type. The benchmark is constructed using a combinatorial design, each factor is varied independently across generated documents, enabling controlled analysis of model behavior. Documents are generated end to end using an LLM pipeline across six layout archetypes, with a 40 percent random override to prevent models from exploiting spurious correlations. Additionally, SynthDocBench spans long-context documents with substantially greater length and structural diversity than existing benchmarks. Evaluating seven frontier VLMs, we uncover three failure modes that existing benchmarks cannot surface: sharp degradation with document length, a systematic positional sensitivity in which the middle third of a document is hardest for five of six models and five of six models show a negative Early-to-Late trend (steepest decline: 8.3 percentage points), and breakdown of chart comprehension in long-document settings. These results suggest that current models may be overfitting to benchmark artifacts rather than achieving robust long-context visual document understanding.
BibTeX
@inproceedings{verma2026synthdocbench,
  author = {Abhigya Verma and Khyati Mahajan and Amit Kumar Saha and Shruthan Radhakrishna and Sagar Davasam and Vikas Yadav and Sai Rajeswar Mudumba},
  title = {{SynthDocBench: Controlled Benchmark for Long-Context Visual Document Understanding}},
  booktitle = {COLM},
  year = {2026},
  url = {https://openreview.net/forum?id=NmjcBUaUqb},
}
Figure