Introduction
Visual Document Understanding (VDU) is a critical task in document AI, involving the extraction of structured information from various document types such as forms, receipts, and reports. While existing pre-training frameworks like the LayoutLM family have significantly advanced the field, they often operate at granularities that are either too coarse (entire images) or too fine (individual words or image patches).
Wukong-Reader introduces a shift in focus toward the textline as the intrinsic granularity for VDU. A textline typically encapsulates semantically and spatially related words—such as a complete address or a full name—providing a more natural unit for alignment than isolated words.
Methodology
The Wukong-Reader framework employs a hybrid architecture combining a dual-stream encoder for initial processing and a single-stream multi-modal Transformer for fusion. The visual stream utilizes a Mask-RCNN backbone, while the textual stream is based on a RoBERTa encoder.
The model is pre-trained using four distinct objectives designed to capture fine-grained cross-modal interactions:
- Textline-Region Contrastive Learning (TRC): This core objective aligns visual representations of textline regions (extracted via an RoIHead) with their corresponding textual content using a contrastive loss.
- Masked Region Modeling (MRM): The model reconstructs pixels within masked textline regions, forcing the visual encoder to learn structural patterns of text.
- Textline Grid Matching (TGM): This task requires the model to predict the grid cell location of specific words, explicitly modeling document layout.
- Masked Language Modeling (MLM): A standard text-based objective enhanced with layout information.
Experimental Results
Wukong-Reader was pre-trained on the IIT-CDIP dataset (11 million document images) and evaluated across several benchmarks:
- FUNSD: The model achieved a state-of-the-art (SOTA) weighted F1 score of 93.62, surpassing LayoutLMv3.
- SROIE: It reached an F1 score of 98.15, establishing a new benchmark for receipt information extraction.
- CORD: The model maintained high competitiveness with an F1 score of 97.27.
Beyond benchmark scores, the authors observed that Masked Region Modeling (MRM) serves as a vital stabilizer during training, preventing numerical instability. Furthermore, the TRC objective enables emergent localization capabilities, allowing the model to map text to visual regions effectively without complex post-processing.
Conclusion and Future Work
Wukong-Reader demonstrates that aligning vision and language at the textline level provides a superior "sweet spot" for document understanding. While the model excels at information extraction, future work will focus on improving performance in Document VQA and exploring end-to-end systems that eliminate the dependency on external OCR engines for textline definition.
Summary updated