PDF-VQA: A New Dataset for Real-World VQA on PDF Documents

Yihao Ding, Siwen Luo, Hyunsuk Chung, Soyeon Caren Han

TL;DR

This paper introduces PDF-VQA, a novel dataset and benchmark for Visual Question Answering (VQA) on PDF documents, extending beyond single pages to full multi-page documents. They also propose LoSpa, a graph-based VQA model that leverages spatial and hierarchical relationships between document elements to achieve state-of-the-art performance.

What Problem Does This Paper Address?

Visual Question Answering (VQA) has seen significant progress, but most existing work focuses on natural images or single-page documents. Understanding complex, multi-page documents like scientific articles, which have intricate layouts and hierarchical structures, remains a significant challenge. Current document VQA datasets often limit questions to textual content or single-page layouts, failing to capture the full context and relationships across multiple pages. This gap hinders the development of AI systems that can truly comprehend and interact with real-world documents.

Methods

The core contribution lies in the PDF-VQA dataset and the LoSpa model.

The PDF-VQA dataset is designed to comprehensively evaluate document understanding by introducing three tasks:

  • Task A (Page-level Document Element Recognition): Focuses on identifying document elements and their relative positions on a single page.
  • Task B (Page-level Document Layout Structure Understanding): Assesses understanding of spatial and logical structures on a page, including information extraction.
  • Task C (Full Document-level Understanding): The most novel aspect, this task extends VQA to multi-page documents, requiring understanding of hierarchical relationships across the entire document.

To facilitate this, the authors provide relational graphs:

  • Logical Relational Graph (LR Graph): Captures parent-child relationships between document elements based on hierarchical structures.
  • Spatial Relational Graph (SR Graph): Encodes relative spatial relationships between elements using their bounding box coordinates.

The proposed LoSpa (Logical and Spatial Graph-based Model) leverages these graphs. It represents document elements as nodes in a graph and uses Graph Convolutional Networks (GCNs) to learn enhanced node representations that incorporate both logical and spatial relationships. These enriched representations are then fed into a transformer-based architecture for question answering.

Datasets

The primary contribution is the PDF-VQA dataset, which is a new benchmark for document-based VQA. It is derived from PDF versions of scientific articles from the PubMed Central (PMC) Open Access Subset. The dataset includes:

  • Document Element Recognition: Bounding boxes and categories for elements like title, text, list, figure, table, table caption, and figure caption.
  • Relational Graphs: Annotated LR and SR graphs for document elements.
  • Question-Answer Pairs: Categorized into three tasks (A, B, and C) covering page-level element recognition, page-level structure understanding, and full document-level understanding, respectively.

The dataset is augmented with automatically generated question-answer pairs using templates to ensure diversity and scale.

Key Insights and Ideas

  • Multi-page understanding is crucial: The PDF-VQA dataset highlights that real-world document understanding often requires processing information across multiple pages, a dimension largely overlooked by previous VQA benchmarks.
  • Relational information is key: Explicitly encoding and leveraging spatial and hierarchical relationships between document elements (via LR and SR graphs) significantly boosts performance in document VQA tasks.
  • Object-level features outperform token-level: The study suggests that using object-level features (representing entire document elements) is more effective for document VQA than token-level features, especially for complex multi-page documents.
  • Graph-based models are powerful: The LoSpa model demonstrates the efficacy of graph neural networks in capturing and utilizing structural information for improved VQA performance.

Key Results

  • LoSpa achieves state-of-the-art performance: The proposed LoSpa model outperforms several strong baselines (including VisualBERT, VILT, LXMERT, and LayoutLM2) across all three tasks of the PDF-VQA dataset.
  • Task C is the most challenging: Performance drops significantly on Task C (full document understanding), indicating the inherent difficulty of multi-page document comprehension and highlighting a substantial area for future research.
  • Relational information provides significant gains: Ablation studies show that incorporating both logical and spatial relations (LoSpa) leads to substantial improvements over models without this information. For instance, LoSpa achieves 94.55% accuracy on Task A, 90.64% on Task B, and 28.99% on Task C (test set).
  • Human evaluation confirms quality: The automatically generated QA pairs were evaluated by human raters and achieved high positive rates for relevance, correctness, and meaningfulness, indicating the dataset's quality.

Limitations and Future Work

  • Task C's difficulty: The paper acknowledges that Task C, focusing on full document understanding, is the most challenging and has the most room for improvement. This suggests future work should focus on developing more sophisticated models for cross-page reasoning and hierarchical understanding.
  • Model limitations: While LoSpa performs well, the performance on Task C is still relatively low, indicating that current approaches may not fully capture the complexities of multi-page document comprehension.
  • Future directions: The authors suggest that their PDF-VQA dataset and the proposed graph-based approach provide a strong foundation for future research in document-based VQA, particularly in developing models that can handle multi-page documents and leverage rich structural information. They also mention the potential for exploring more advanced graph neural network architectures and attention mechanisms.

Summary updated

Abstract

Document-based Visual Question Answering examines the document understanding of document images in conditions of natural language questions. We proposed a new document-based VQA dataset, PDF-VQA, to comprehensively examine the document understanding from various aspects, including document element recognition, document layout structural understanding as well as contextual understanding and key information extraction. Our PDF-VQA dataset extends the current scale of document understanding that limits on the single document page to the new scale that asks questions over the full document of multiple pages. We also propose a new graph-based VQA model that explicitly integrates the spatial and hierarchically structural relationships between different document elements to boost the document structural understanding. The performances are compared with several baselines over different question types and tasks (The full dataset is released in https://github.com/adlnlp/pdfvqa).
BibTeX
@inproceedings{ding2023pdf,
  author = {Yihao Ding and Siwen Luo and Hyunsuk Chung and Soyeon Caren Han},
  title = {{PDF-VQA: A New Dataset for Real-World VQA on PDF Documents}},
  booktitle = {ECML/PKDD},
  year = {2023},
  url = {https://openreview.net/forum?id=YODwDhx2Nq},
}
Figure