TL;DR
ViDoRe V3 is a comprehensive multimodal Retrieval-Augmented Generation (RAG) benchmark featuring 26,000 visually rich document pages and over 3,000 human-verified queries across 10 domains and 6 languages. It evaluates the three core pillars of modern RAG: retrieval, answer generation, and visual grounding. The benchmark reveals that while visual retrievers and hybrid contexts excel at handling complex layouts, fine-grained visual grounding remains a massive bottleneck for state-of-the-art models.
What Problem Does This Paper Address?
While Retrieval-Augmented Generation (RAG) has become the go-to architecture for knowledge-intensive AI tasks, standard academic benchmarks oversimplify real-world enterprise data. Real documents—like financial reports, textbooks, and manuals—are visually rich, containing charts, tables, and infographics that text-only pipelines completely ignore.
Furthermore, existing benchmarks often evaluate retrieval or single-page comprehension in isolation, failing to capture the multi-page, cross-document synthesis required in practice. Finally, to build trustworthy systems that mitigate hallucinations, models must be able to ground their answers in specific document regions (using bounding boxes), a capability poorly tracked by current evaluation datasets.
Methods
The authors developed a rigorous three-stage human-in-the-loop pipeline to build ViDoRe V3:
- Context Preparation: Document text and images are extracted using parsing tools, and both single-section and cross-section summaries are generated to serve as context.
- Query Generation: The dataset utilizes a dual-axis taxonomy combining 7 Query Types (open-ended, extractive, numerical, multi-hop, compare-contrast, boolean, enumerative) and 3 Query Formats (question, keyword, instruction). Queries are generated via both automated LLM pipelines and human experts.
- Grounded Query Answering: To scale human annotation efficiently, a state-of-the-art Vision-Language Model (VLM), Qwen2.5-VL-32B, pre-filters candidate pages. Human experts then verify page relevance, draw bounding boxes around supporting evidence, and write answers. A final VLM aggregation step synthesizes these into a single "golden" reference answer.
Datasets
The benchmark introduces ViDoRe V3, which comprises 10 diverse, industry-relevant public and private corpora (spanning Finance, Computer Science, Energy, Pharmaceuticals, and HR). It contains:
- 26,000 document pages available in image, text, and PDF formats.
- 3,099 human-verified queries, each translated into 6 languages (English, French, Spanish, German, Italian, and Portuguese) to rigorously evaluate cross-lingual retrieval.
- To prevent data contamination, 8 datasets are publicly released, while 2 are held back as a private test set for blind evaluation.
Key Insights and Ideas
- Vision Wins Retrieval, Text Wins Reranking: Visual retrievers naturally handle complex document layouts better than text-only models. However, current visual rerankers are surprisingly weak. Adding a textual reranker to a pipeline yields massive performance gains, whereas visual rerankers offer marginal improvements or even degrade performance.
- Mixed Modalities are the Hardest: Document pages that integrate multiple modalities (e.g., text wrapped around a chart) pose the greatest challenge for retrievers. Models struggle to synthesize information when text and visuals are tightly intertwined on a single page.
- Grounding is a Massive Blind Spot: State-of-the-art VLMs are remarkably poor at pinpointing where they found an answer. Their bounding box accuracy is an order of magnitude lower than human agreement, highlighting a critical gap in spatial document understanding.
Key Results
- Retrieval Performance: Visual retrievers consistently outperformed textual ones. ColEmbed-3B-v2 was the top-performing retriever with an average NDCG@10 of 59.8. However, performance drops sharply as the number of relevant pages increases, proving that multi-page synthesis is a major bottleneck.
- Modality Impact: Performance was highest on plain text pages (59.3 NDCG@10) and lowest on pages with mixed content types (45.1 NDCG@10).
- Generation Quality: Providing image-based context to Gemini 3 Pro outperformed text-only context on hard queries by up to 2.8 percentage points. A hybrid pipeline combining the top text and image retrieval results yielded the highest overall accuracy on challenging queries (54.7%).
- Visual Grounding: Human inter-annotator agreement on bounding boxes achieved an F1 score of 0.602. Meanwhile, the best-performing model, Qwen3-VL-30B, achieved a dismal F1 score of just 0.089, frequently failing to annotate pages that humans flagged as containing evidence.
Limitations and Future Work
- Language and Script Constraints: While the queries are translated into 6 languages, the source documents are restricted to English and French. Future iterations need to include non-Latin scripts and lower-resource language families.
- Document Distribution Bias: The benchmark focuses on clean, public, long-form documents. It does not account for noisy, short-form enterprise data like emails, support tickets, or handwritten notes.
- Annotation Subjectivity: Open-ended reasoning and visual grounding are inherently subjective. The authors note that multiple valid retrieval paths and bounding box granularities can exist beyond their annotated ground truths.
Summary updated