ViDoRe V3: A Comprehensive Evaluation of Retrieval Augmented Generation in Complex Real-World Scenarios

António Loison, Quentin Macé, Antoine Edy, Victor Xing, Tom Balough, Gabriel Moreira, Bo Liu, Manuel Faysse, Céline Hudelot, Gautier Viaud

TL;DR

ViDoRe V3 is a comprehensive multimodal Retrieval-Augmented Generation (RAG) benchmark featuring 26,000 visually rich document pages and over 3,000 human-verified queries across 10 domains and 6 languages. It evaluates the three core pillars of modern RAG: retrieval, answer generation, and visual grounding. The benchmark reveals that while visual retrievers and hybrid contexts excel at handling complex layouts, fine-grained visual grounding remains a massive bottleneck for state-of-the-art models.

What Problem Does This Paper Address?

While Retrieval-Augmented Generation (RAG) has become the go-to architecture for knowledge-intensive AI tasks, standard academic benchmarks oversimplify real-world enterprise data. Real documents—like financial reports, textbooks, and manuals—are visually rich, containing charts, tables, and infographics that text-only pipelines completely ignore.

Furthermore, existing benchmarks often evaluate retrieval or single-page comprehension in isolation, failing to capture the multi-page, cross-document synthesis required in practice. Finally, to build trustworthy systems that mitigate hallucinations, models must be able to ground their answers in specific document regions (using bounding boxes), a capability poorly tracked by current evaluation datasets.

Methods

The authors developed a rigorous three-stage human-in-the-loop pipeline to build ViDoRe V3:

Figure 2: Overview of the benchmark creation process. Queries are sourced from 3 streams: human extractive (using raw pages), human blind contextual (using summaries to mitigate extractive bias), and synthetic blind contextual. For each query, a VLM pre-filtered subset of candidate pages is labeled by 1–3 human annotators that perform relevance scoring, bounding box localization and answer generation. A final response aggregation combines annotator answers into a single answer.
  1. Context Preparation: Document text and images are extracted using parsing tools, and both single-section and cross-section summaries are generated to serve as context.
  2. Query Generation: The dataset utilizes a dual-axis taxonomy combining 7 Query Types (open-ended, extractive, numerical, multi-hop, compare-contrast, boolean, enumerative) and 3 Query Formats (question, keyword, instruction). Queries are generated via both automated LLM pipelines and human experts.
  3. Grounded Query Answering: To scale human annotation efficiently, a state-of-the-art Vision-Language Model (VLM), Qwen2.5-VL-32B, pre-filters candidate pages. Human experts then verify page relevance, draw bounding boxes around supporting evidence, and write answers. A final VLM aggregation step synthesizes these into a single "golden" reference answer.

Datasets

The benchmark introduces ViDoRe V3, which comprises 10 diverse, industry-relevant public and private corpora (spanning Finance, Computer Science, Energy, Pharmaceuticals, and HR). It contains:

  • 26,000 document pages available in image, text, and PDF formats.
  • 3,099 human-verified queries, each translated into 6 languages (English, French, Spanish, German, Italian, and Portuguese) to rigorously evaluate cross-lingual retrieval.
  • To prevent data contamination, 8 datasets are publicly released, while 2 are held back as a private test set for blind evaluation.

Key Insights and Ideas

  • Vision Wins Retrieval, Text Wins Reranking: Visual retrievers naturally handle complex document layouts better than text-only models. However, current visual rerankers are surprisingly weak. Adding a textual reranker to a pipeline yields massive performance gains, whereas visual rerankers offer marginal improvements or even degrade performance.
  • Mixed Modalities are the Hardest: Document pages that integrate multiple modalities (e.g., text wrapped around a chart) pose the greatest challenge for retrievers. Models struggle to synthesize information when text and visuals are tightly intertwined on a single page.
  • Grounding is a Massive Blind Spot: State-of-the-art VLMs are remarkably poor at pinpointing where they found an answer. Their bounding box accuracy is an order of magnitude lower than human agreement, highlighting a critical gap in spatial document understanding.

Key Results

  • Retrieval Performance: Visual retrievers consistently outperformed textual ones. ColEmbed-3B-v2 was the top-performing retriever with an average NDCG@10 of 59.8. However, performance drops sharply as the number of relevant pages increases, proving that multi-page synthesis is a major bottleneck.
Figure 5: ColEmbed-3B-v2 NDCG@10 by query type and format.
  • Modality Impact: Performance was highest on plain text pages (59.3 NDCG@10) and lowest on pages with mixed content types (45.1 NDCG@10).
Figure 6: ColEmbed-3B-v2 NDCG@10 by modality.
  • Generation Quality: Providing image-based context to Gemini 3 Pro outperformed text-only context on hard queries by up to 2.8 percentage points. A hybrid pipeline combining the top text and image retrieval results yielded the highest overall accuracy on challenging queries (54.7%).
  • Visual Grounding: Human inter-annotator agreement on bounding boxes achieved an F1 score of 0.602. Meanwhile, the best-performing model, Qwen3-VL-30B, achieved a dismal F1 score of just 0.089, frequently failing to annotate pages that humans flagged as containing evidence.

Limitations and Future Work

  • Language and Script Constraints: While the queries are translated into 6 languages, the source documents are restricted to English and French. Future iterations need to include non-Latin scripts and lower-resource language families.
  • Document Distribution Bias: The benchmark focuses on clean, public, long-form documents. It does not account for noisy, short-form enterprise data like emails, support tickets, or handwritten notes.
  • Annotation Subjectivity: Open-ended reasoning and visual grounding are inherently subjective. The authors note that multiple valid retrieval paths and bounding box granularities can exist beyond their annotated ground truths.

Summary updated

Abstract

Retrieval-Augmented Generation (RAG) pipelines must address challenges beyond simple single-document retrieval, such as interpreting visual elements (tables, charts, images), synthesizing information across documents, and providing accurate source grounding. Existing benchmarks fail to capture this complexity, often focusing on textual data, single-document comprehension, or evaluating retrieval and generation in isolation. We introduce ViDoRe v3, a comprehensive multimodal RAG benchmark featuring multi-type queries over visually rich document corpora. It covers 10 datasets across diverse professional domains, comprising ~26,000 document pages paired with 3,099 human-verified queries, each available in 6 languages. Through 12,000 hours of human annotation effort, we provide high-quality annotations for retrieval relevance, bounding box localization, and verified reference answers. Our evaluation of state-of-the-art RAG pipelines reveals that visual retrievers outperform textual ones, late-interaction models and textual reranking substantially improve performance, and hybrid or purely visual contexts enhance answer generation quality. However, current models still struggle with non-textual elements, open-ended queries, and fine-grained visual grounding. To encourage progress in addressing these challenges, the benchmark is released under a commercially permissive license at https://hf.co/vidore.
BibTeX
@inproceedings{loison2026vidore,
  author = {António Loison and Quentin Macé and Antoine Edy and Victor Xing and Tom Balough and Gabriel Moreira and Bo Liu and Manuel Faysse and Céline Hudelot and Gautier Viaud},
  title = {{ViDoRe V3: A Comprehensive Evaluation of Retrieval Augmented Generation in Complex Real-World Scenarios}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.755},
}
Figure