Abstract
VisualMRC is a novel dataset and model architecture designed to advance Machine Reading Comprehension (MRC) by integrating text with visual layout information. By utilizing 30,000+ question-answer pairs from web document images, the authors demonstrate that spatial and semantic layout cues significantly enhance a model's ability to generate natural language answers.
Introduction
Most Machine Reading Comprehension (MRC) research treats documents as "plain text," ignoring the rich information found in visual layouts like headers, sidebars, tables, and image captions. Conversely, Visual Question Answering (VQA) typically focuses on natural scenes rather than dense, text-heavy documents. There is a significant gap in how AI handles "born-digital" documents like webpages or PDFs. To answer a question about a webpage, humans use font size to identify titles, position to find captions, and structure to parse lists. VisualMRC bridges this gap, moving the field toward agents that can comprehend real-world document images.
The VisualMRC Dataset
The primary contribution of this work is the VisualMRC dataset, which contains 10,197 document images sourced from 35 webpage domains, featuring 30,562 question-answer pairs. Unlike previous datasets, it provides abstractive answers (full sentences generated by humans) rather than just pointing to a snippet of text.
As shown in the dataset analysis, VisualMRC features longer and more complex questions and answers compared to existing benchmarks like TextVQA and DocVQA. The word clouds highlight the diversity of topics and vocabulary present in the dataset.
Methodology: Layout-Aware Transformers
The authors propose a "Layout-aware" sequence-to-sequence model based on pre-trained Transformers (specifically BART and T5). Their approach involves several novel components:
- Visual Input Integration: Instead of just feeding text tokens to the encoder, they inject three additional types of embeddings for every OCR-detected word:
- Segment Embeddings: Identifies the semantic class of the region (e.g., Heading, Paragraph, or List).
- Location Embeddings: Encodes the relative (x, y) coordinates of the word's bounding box.
- Appearance Embeddings: Uses a Faster R-CNN to extract visual features from the region.
- Saliency Detection: A sub-module predicts which tokens are most relevant to the question, filtering out "noise" common in complex webpage layouts.
- Multi-task Learning: The model is trained simultaneously to generate the correct answer and to identify the "salient" regions of the document.
This example illustrates how document regions are segmented into semantic categories, which serves as a critical signal for the model to understand the document's hierarchy and structure.
Key Results
The proposed LayoutT5 and LayoutBART models outperformed state-of-the-art VQA models (like M4C) and text-only baselines across all metrics (BLEU, METEOR, ROUGE-L, and CIDEr).
- Quantitative Boost: Adding layout information (segment, location, and appearance) to a T5 model improved its CIDEr score from 318.6 to 335.1.
- Human Gap: While the models are impressive, a significant gap remains. On the CIDEr metric, the best model scored 344.1, while humans scored 370.3, suggesting that models still struggle with nuanced reasoning on visual data.
Conclusion
VisualMRC demonstrates that document structure is a critical semantic signal. While ROI detection remains a bottleneck, the integration of layout-aware embeddings represents a significant step forward for document intelligence. Future work will likely focus on better pre-training for layouts and deeper understanding of non-text objects like charts.
Summary updated