Abstract
We present MATrIX - a Modality-Aware Transformer for Information eXtraction in the Visual Document Understanding (VDU) domain. VDU covers information extraction from visually rich documents such as forms, invoices, receipts, tables, graphs, presentations, or advertisements. In these, text semantics and visual information supplement each other to provide a global understanding of the document. MATrIX is pre-trained in an unsupervised way with specifically designed tasks that require the use of multi-modal information (spatial, visual, or textual). We consider the spatial and text modalities all at once in a single token set. To make the attention more flexible, we use a learned modality-aware relative bias in the attention mechanism to modulate the attention between the tokens of different modalities. We evaluate MATrIX on 3 different datasets each with strong baselines.
Introduction
Visual Document Understanding (VDU) is a critical frontier in automated document processing. Unlike standard natural language processing, VDU requires models to interpret text within its spatial and visual context. Existing approaches often suffer from two main drawbacks: rigid fusion of modalities and strict context limits due to sub-word tokenization. MATrIX addresses these challenges by introducing a flexible modality-aware attention mechanism and a word-level tokenization strategy that significantly extends the effective context window.
Methods and Architecture
The MATrIX architecture is designed to treat visual and textual information as a unified set of tokens while maintaining awareness of their distinct modalities through specialized embedding layers.
Modality-Aware Relative Attention
The core innovation of MATrIX is its attention mechanism. Instead of treating all token interactions equally, it employs a learned modality-aware relative bias. This bias is conditioned on the specific pair of modalities involved in the attention calculation (e.g., text-to-text, text-to-vision, or vision-to-vision). This allows the model to dynamically adjust how it weights semantic flow versus spatial alignment depending on the tokens it is comparing.
Word-Level Representations
To overcome the 512-token limit common in transformers, MATrIX shifts from sub-word to word-level representations. By averaging sub-word embeddings into a single word token, the model can process approximately double the amount of text compared to traditional sub-word models. This allows it to "see" much longer documents without the need for truncation, which often discards vital information.
Pre-training Tasks
MATrIX utilizes several unsupervised pre-training tasks to learn complex cross-modal relationships:
- Masked Language Modeling (MLM): Standard text reconstruction from context.
- Line Bounding Box Regression: Predicting the coordinates of an entire line based on a single word token. This task was found to be the most impactful, providing a +1.72 F1 boost by teaching the model document structure.
- Learn to Reconstruct: Attempting to recreate the original image from the tokens.
- Line Redaction and Token Switching: Additional tasks designed to strengthen spatial and semantic understanding.
Experimental Results
MATrIX was pre-trained on 5 million pages from the IIT-CDIP collection and evaluated on three major benchmarks:
- CORD (Receipts): Achieved a competitive 96.05 F1 score, nearly matching state-of-the-art models while using 3x fewer parameters than heavyweights like LayoutLMv2-large.
- RVL-CDIP (Classification): Reached 94.20% accuracy, demonstrating strong performance in large-scale document classification on par with much larger multi-modal models.
- FUNSD (Forms): Achieved an F1 of 78.60. While slightly below some SOTA models, this highlights the challenge of small-scale datasets for complex modality-aware biases.
Ablation studies confirmed that the spatial relative attention bias was the single most significant contributor to performance, yielding an 11.62 F1 improvement.
Conclusion
MATrIX demonstrates that modality-aware attention and word-level tokenization can lead to highly efficient and effective document understanding. By focusing on how different modalities interact and extending the context window, MATrIX provides a robust framework for VDU tasks with a significantly reduced parameter count. Future work will explore more complex structural extraction and entity-relationship modeling, such as linking labels to their specific values in complex forms.
Summary updated