Wukong-Reader: Multi-modal Pre-training for Fine-grained Visual Document Understanding

Haoli Bai, Zhiguang Liu, Xiaojun Meng, Wentao Li, Shuang Liu, Yifeng Luo, Nian Xie, Rongfu Zheng, Liangwei Wang, Lu Hou, Jiansheng Wei, Xin Jiang, Qun Liu

Visual Document Understanding (VDU) is a critical field in AI, enabling machines to interpret complex documents like forms, receipts, and reports. Traditional approaches typically process documents at the level of individual words, image patches, or detected objects. However, these granularities often overlook the inherent structural logic of documents: the textline. In their ACL 2023 paper, researchers from Huawei’s Noah’s Ark Lab present Wukong-Reader, a framework that leverages textlines as a fine-grained semantic anchor for multi-modal pre-training.

Examples of textline-level granularity in (a) letters and (b) receipts.

The Textline Advantage

Most state-of-the-art VDU models, such as LayoutLMv3, align visual and textual information at the word or patch level. While effective, this can lead to a loss of semantic coherence. In real-world documents, meaningful information—such as a "Total Amount" or a "Shipping Address"—is typically contained within a single textline. By failing to explicitly align visual regions with these semantic spans, models may miss crucial context regarding document construction. Wukong-Reader addresses this by treating the textline as the fundamental unit of meaning, bridging the gap between global page layouts and local word features.

Methodology

Wukong-Reader employs a hybrid architecture consisting of a dual-stream setup (independent image and text encoders) followed by a single-stream multi-modal encoder for feature fusion.

The Wukong-Reader architecture and its multi-modal pre-training objectives.

The core innovation lies in three novel pre-training objectives designed to exploit textline structures:

  1. Textline-Region Contrastive Learning (TRC): This task aligns visual features of a textline (extracted via an RoIHead) with the textual features of the constituent words. By utilizing a maximum similarity metric, the model learns precise correspondences between visual regions and text strings.
  2. Masked Region Modeling (MRM): The model masks 15% of textline regions in the document image and attempts to reconstruct the original pixels. This objective serves as a powerful regularizer, stabilizing training and allowing for higher learning rates without the numerical instabilities often encountered in multi-modal masking.
  3. Textline Grid Matching (TGM): The document image is partitioned into a grid (e.g., 7x7). The model is tasked with predicting which grid cell a specific word belongs to without access to the word's 2D bounding box. This forces the model to rely on visual cues and surrounding context for localization.

Experimental Evaluation

The model was pre-trained on the IIT-CDIP dataset (11 million English document images) and a newly curated Chinese corpus of 8 million documents. Evaluation across a diverse set of VDU benchmarks demonstrated the model's superiority:

  • Information Extraction: Wukong-Reader achieved a weighted F1 score of 93.62 on FUNSD (Large model), outperforming LayoutLMv3 (92.08). On the SROIE receipt dataset, it set a new record with an F1 score of 98.15.
  • Multilingual Performance: On the XFUND-RE task, the model reached 85.82%, marking a significant improvement over previous multilingual baselines.
  • Efficiency: Despite its performance gains, Wukong-Reader maintains high inference efficiency. The Large model records a latency of 794ms, compared to 964ms for LayoutLMv3, due to its optimized handling of visual tokens.

Discussion and Limitations

A key insight from the study is that textlines act as stable semantic anchors, providing a natural bridge for multi-modal alignment. Furthermore, the inclusion of visual reconstruction (MRM) was found to be essential for training stability. However, the current iteration of Wukong-Reader is limited by a fixed sequence length of 512 tokens, which presents challenges for multi-page or exceptionally long documents. Additionally, while the model excels at discriminative tasks like extraction and classification, it is not optimized for generative tasks such as document summarization.

Conclusion

Wukong-Reader introduces a novel paradigm for VDU by focusing on textline-level granularity. By aligning visual regions with semantic text spans through innovative pre-training tasks, it achieves state-of-the-art results while improving inference speed. Future work will explore relative positional embeddings and architectural adjustments to better handle long-form and multi-page document structures.

Summary updated

Abstract

Unsupervised pre-training on millions of digital-born or scanned documents has shown promising advances in visual document understanding (VDU). While various vision-language pre-training objectives are studied in existing solutions, the document textline, as an intrinsic granularity in VDU, has seldom been explored so far. A document textline usually contains words that are spatially and semantically correlated, which can be easily obtained from OCR engines. In this paper, we propose Wukong-Reader, trained with new pre-training objectives to leverage the structural knowledge nested in document textlines. We introduce textline-region contrastive learning to achieve fine-grained alignment between the visual regions and texts of document textlines. Furthermore, masked region modeling and textline-grid matching are also designed to enhance the visual and layout representations of textlines. Experiments show that Wukong-Reader brings superior performance on various VDU tasks in both English and Chinese. The fine-grained alignment over textlines also empowers Wukong-Reader with promising localization ability.
BibTeX
@inproceedings{bai2023wukong,
  author = {Haoli Bai and Zhiguang Liu and Xiaojun Meng and Wentao Li and Shuang Liu and Yifeng Luo and Nian Xie and Rongfu Zheng and Liangwei Wang and Lu Hou and Jiansheng Wei and Xin Jiang and Qun Liu},
  title = {{Wukong-Reader: Multi-modal Pre-training for Fine-grained Visual Document Understanding}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.acl-long.748},
}
Figure