VisualMRC: Machine Reading Comprehension on Document Images

Ryota Tanaka, Kyosuke Nishida, Sen Yoshida

Abstract

Recent studies on machine reading comprehension have focused on text-level understanding but have not yet reached the level of human understanding of the visual layout and content of real-world documents. In this study, we introduce a new visual machine reading comprehension dataset, named VisualMRC, wherein given a question and a document image, a machine reads and comprehends texts in the image to answer the question in natural language. Compared with existing visual question answering (VQA) datasets that contain texts in images, VisualMRC focuses more on developing natural language understanding and generation abilities. It contains 30,000+ pairs of a question and an abstractive answer for 10,000+ document images sourced from multiple domains of webpages. We also introduce a new model that extends existing sequence-to-sequence models, pre-trained with large-scale text corpora, to take into account the visual layout and content of documents. Experiments with VisualMRC show that this model outperformed the base sequence-to-sequence models and a state-of-the-art VQA model. However, its performance is still below that of humans on most automatic evaluation metrics. The dataset will facilitate research aimed at connecting vision and language understanding.

Introduction

Standard Machine Reading Comprehension (MRC) models are typically designed to process plain text. However, real-world documents—such as webpages, PDFs, and presentations—are inherently multimodal. They rely on visual layouts, headers, lists, and charts to convey meaning. While existing Visual Question Answering (VQA) datasets like TextVQA or DocVQA address images containing text, they often focus on short, extractive answers. They lack the requirement for deep natural language understanding (NLU) and the generation of complex, abstractive answers (NLG) based on intricate document structures. To bridge this gap, we introduce VisualMRC, a task requiring models to visually "read" a document, understand its structural layout, and generate conversational answers to open-domain questions.

The VisualMRC Dataset

The VisualMRC dataset consists of 10,197 document images sourced from 35 webpage domains, paired with 30,562 human-annotated question-answer pairs. A key feature of this dataset is the inclusion of Region-of-Interest (ROI) annotations, where text blocks are categorized into nine semantic classes, such as Heading, Paragraph, List, and Data.

Figure 3: Example of a document image in VisualMRC with Region-of-Interest (ROI) annotations across different semantic classes.

Compared to datasets like TextVQA and DocVQA, VisualMRC presents a significantly higher linguistic complexity. Document images in VisualMRC contain a much larger number of words, and the answers are abstractive, averaging 9.53 words in length. This requires models to synthesize information across the entire document rather than simply locating a single word or phrase.

Figure 4: Distribution of word counts in document images across VisualMRC, TextVQA, and DocVQA datasets, highlighting the increased complexity of VisualMRC.

Methodology: LayoutBART and LayoutT5

We propose extending powerful pre-trained sequence-to-sequence models, specifically BART and T5, to incorporate visual and spatial information. Our modified architectures, LayoutBART and LayoutT5, introduce several innovations:

  • ROI Tokens: Input sequences include special tokens representing the semantic class of each text block (e.g., [Title], [Paragraph]).
  • Multimodal Embeddings: We fuse standard token embeddings with segment embeddings (ROI class), location embeddings (4D bounding box coordinates), and appearance embeddings (visual features extracted via Faster R-CNN).
  • Auxiliary Saliency Detection: To manage the noise of dense OCR text, the model is trained to predict a "saliency" score for each token, identifying which words are most relevant to the given question.

Experimental Results

In end-to-end evaluations, LayoutT5 achieved a BLEU-4 of 41.0 and a ROUGE-L of 52.2, significantly outperforming the state-of-the-art VQA model M4C (10.2 BLEU-4). Ablation studies confirmed that each multimodal component—ROI labels, location, and appearance embeddings—contributed to incremental performance gains over text-only baselines.

Despite these advancements, a notable gap remains between machine and human performance. Humans achieve a ROUGE-L of 57.9 and a CIDEr of 370.3, compared to LayoutBART's 55.2 ROUGE-L and 329.1 CIDEr. This suggests that while models are gaining visual awareness, they still struggle to fully synthesize complex document data as effectively as humans.

Conclusion

VisualMRC establishes a new frontier for machine reading comprehension by emphasizing the role of document layout in natural language generation. While our Layout-aware models show promising results, future work is needed to improve ROI detection pipelines and develop pre-training methods specifically tailored for vision-and-language tasks in structured documents.

Summary updated

Abstract

Recent studies on machine reading comprehension have focused on text-level understanding but have not yet reached the level of human understanding of the visual layout and content of real-world documents. In this study, we introduce a new visual machine reading comprehension dataset, named VisualMRC, wherein given a question and a document image, a machine reads and comprehends texts in the image to answer the question in natural language. Compared with existing visual question answering (VQA) datasets that contain texts in images, VisualMRC focuses more on developing natural language understanding and generation abilities. It contains 30,000+ pairs of a question and an abstractive answer for 10,000+ document images sourced from multiple domains of webpages. We also introduce a new model that extends existing sequence-to-sequence models, pre-trained with large-scale text corpora, to take into account the visual layout and content of documents. Experiments with VisualMRC show that this model outperformed the base sequence-to-sequence models and a state-of-the-art VQA model. However, its performance is still below that of humans on most automatic evaluation metrics. The dataset will facilitate research aimed at connecting vision and language understanding.
BibTeX
@inproceedings{tanaka2021visualmrc,
  author = {Ryota Tanaka and Kyosuke Nishida and Sen Yoshida},
  title = {{VisualMRC: Machine Reading Comprehension on Document Images}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i15.17635},
}

Loading related content...

Figure