Abstract
DocHop-QA is a large-scale benchmark designed to advance AI capabilities in complex, multi-hop reasoning across multimodal scientific documents. While traditional QA datasets often rely on single-paragraph text matching, DocHop-QA requires models to synthesize information scattered across multiple documents, incorporating text, tables, and visual layouts. Sourced from over 2 million PubMed Central articles, the dataset contains 11,379 QA instances. Our evaluation of state-of-the-art models reveals that even advanced Vision-Language Models (VLMs) struggle with the "alignment tax" and long-context constraints inherent in multi-document, visually rich environments.
Introduction
Most current Question Answering (QA) benchmarks are "shallow," typically requiring models to locate an answer within a single paragraph or document through simple keyword matching. However, real-world scientific inquiry often demands synthesizing information from disparate sources—for instance, connecting a methodology described in one paper to results presented in a table in another.
Existing multi-hop datasets like HotpotQA are largely restricted to plain text and rely on explicit hyperlinks. They fail to capture the complexity of Visually Rich Documents (VRDs), where structural cues like multi-column layouts and tables are essential. DocHop-QA addresses this gap by providing a multimodal, multi-document benchmark that mimics the actual workflow of a human researcher.
Methodology
The authors developed a scalable, LLM-driven pipeline to generate high-quality QA pairs without the prohibitive cost of manual annotation. The process begins with defining 11 "reasoning concepts" (e.g., Comparison, Table-Referencing) that reflect common scientific inquiry patterns.
Document selection is performed using semantic similarity (TF-IDF and the YAKE algorithm) rather than hyperlinks, creating a more realistic "open-ended" reasoning scenario. A fine-tuned Qwen2-7B-Instruct model then generates questions based on these concepts. Answers are matched using two patterns:
- Fan-hop: Aggregating semantically related snippets from multiple documents.
- Chain-hop: Following a logical path, such as a text passage in one document referring to a table in another.
Dataset Characteristics
The resulting DocHop-QA dataset consists of 11,379 QA instances derived from approximately 2 million full-text scientific articles in PubMed Central (PMC).
The dataset is unique in its combination of multi-hop reasoning with multimodal evidence across multiple documents. Approximately 75.3% of the instances are paragraph-oriented, while 24.7% are table-oriented, ensuring a robust test of a model's ability to handle diverse data modalities.
Key Insights
- Layout as a Core Component: Visual layout is not merely auxiliary; it is fundamental to how humans navigate complex documents.
- The "Alignment Tax": Experimental results showed that adding document images sometimes decreased model performance. This suggests that current VLMs struggle to map text to its visual location when processing dozens of pages simultaneously.
- Reasoning Patterns: By distinguishing between Fan-hop (fact aggregation) and Chain-hop (sequential logic), the dataset provides a nuanced view of model capabilities.
Experimental Results
Models were evaluated across several tasks, including evidence localization and generative answering:
- Evidence Localization: LayoutLMv3 achieved a modest 12.35 F1 score in bounding box extraction, highlighting the difficulty of pinpointing evidence in complex layouts.
- Generative Answering: Qwen2.5-VL emerged as a top performer, achieving a BLEU score of 21.56 and a ROUGE-L of 23.90 in free-form generation.
- Human Validation: A study with 15 annotators confirmed that the LLM-generated questions were natural and meaningful, scoring high on reality and fluency.
Conclusion and Future Work
DocHop-QA exposes significant limitations in current AI models, particularly regarding context window constraints and cross-modal fusion. While the current dataset is biomedical, the generation methodology is domain-agnostic and could be applied to legal or financial sectors. Future work will focus on more sophisticated fusion strategies to master visually rich, multi-document reasoning.
Summary updated