VisualMRC: Machine Reading Comprehension on Document Images

Ryota Tanaka, Kyosuke Nishida, Sen Yoshida

Abstract

VisualMRC is a novel dataset and model architecture designed to advance Machine Reading Comprehension (MRC) by integrating text with visual layout information. By utilizing 30,000+ question-answer pairs from web document images, the authors demonstrate that spatial and semantic layout cues significantly enhance a model's ability to generate natural language answers.

Introduction

Most Machine Reading Comprehension (MRC) research treats documents as "plain text," ignoring the rich information found in visual layouts like headers, sidebars, tables, and image captions. Conversely, Visual Question Answering (VQA) typically focuses on natural scenes rather than dense, text-heavy documents. There is a significant gap in how AI handles "born-digital" documents like webpages or PDFs. To answer a question about a webpage, humans use font size to identify titles, position to find captions, and structure to parse lists. VisualMRC bridges this gap, moving the field toward agents that can comprehend real-world document images.

The VisualMRC Dataset

The primary contribution of this work is the VisualMRC dataset, which contains 10,197 document images sourced from 35 webpage domains, featuring 30,562 question-answer pairs. Unlike previous datasets, it provides abstractive answers (full sentences generated by humans) rather than just pointing to a snippet of text.

Dataset statistics including question/answer lengths and word clouds for VisualMRC compared to TextVQA and DocVQA.

As shown in the dataset analysis, VisualMRC features longer and more complex questions and answers compared to existing benchmarks like TextVQA and DocVQA. The word clouds highlight the diversity of topics and vocabulary present in the dataset.

Methodology: Layout-Aware Transformers

The authors propose a "Layout-aware" sequence-to-sequence model based on pre-trained Transformers (specifically BART and T5). Their approach involves several novel components:

  1. Visual Input Integration: Instead of just feeding text tokens to the encoder, they inject three additional types of embeddings for every OCR-detected word:
  2. Segment Embeddings: Identifies the semantic class of the region (e.g., Heading, Paragraph, or List).
  3. Location Embeddings: Encodes the relative (x, y) coordinates of the word's bounding box.
  4. Appearance Embeddings: Uses a Faster R-CNN to extract visual features from the region.
  5. Saliency Detection: A sub-module predicts which tokens are most relevant to the question, filtering out "noise" common in complex webpage layouts.
  6. Multi-task Learning: The model is trained simultaneously to generate the correct answer and to identify the "salient" regions of the document.
Example of a document image from the VisualMRC dataset with annotated semantic regions.

This example illustrates how document regions are segmented into semantic categories, which serves as a critical signal for the model to understand the document's hierarchy and structure.

Key Results

The proposed LayoutT5 and LayoutBART models outperformed state-of-the-art VQA models (like M4C) and text-only baselines across all metrics (BLEU, METEOR, ROUGE-L, and CIDEr).

  • Quantitative Boost: Adding layout information (segment, location, and appearance) to a T5 model improved its CIDEr score from 318.6 to 335.1.
  • Human Gap: While the models are impressive, a significant gap remains. On the CIDEr metric, the best model scored 344.1, while humans scored 370.3, suggesting that models still struggle with nuanced reasoning on visual data.

Conclusion

VisualMRC demonstrates that document structure is a critical semantic signal. While ROI detection remains a bottleneck, the integration of layout-aware embeddings represents a significant step forward for document intelligence. Future work will likely focus on better pre-training for layouts and deeper understanding of non-text objects like charts.

Summary updated

Abstract

Recent studies on machine reading comprehension have focused on text-level understanding but have not yet reached the level of human understanding of the visual layout and content of real-world documents. In this study, we introduce a new visual machine reading comprehension dataset, named VisualMRC, wherein given a question and a document image, a machine reads and comprehends texts in the image to answer the question in natural language. Compared with existing visual question answering (VQA) datasets that contain texts in images, VisualMRC focuses more on developing natural language understanding and generation abilities. It contains 30,000+ pairs of a question and an abstractive answer for 10,000+ document images sourced from multiple domains of webpages. We also introduce a new model that extends existing sequence-to-sequence models, pre-trained with large-scale text corpora, to take into account the visual layout and content of documents. Experiments with VisualMRC show that this model outperformed the base sequence-to-sequence models and a state-of-the-art VQA model. However, its performance is still below that of humans on most automatic evaluation metrics. The dataset will facilitate research aimed at connecting vision and language understanding.
BibTeX
@inproceedings{tanaka2021visualmrc,
  author = {Ryota Tanaka and Kyosuke Nishida and Sen Yoshida},
  title = {{VisualMRC: Machine Reading Comprehension on Document Images}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i15.17635},
}
Figure