USTC-iFLYTEK at DocILE: A Multi-modal Approach Using Domain-specific GraphDoc

Yan Wang, Jun Du, Jiefeng Ma, Pengfei Hu, Zhenrong Zhang, Jianshu ZhangCLEF (Working Notes) 2023 · OpenReview

Introduction

The Document Information Localization and Extraction (DocILE) 2023 challenge presents a significant hurdle in Document AI: extracting structured data from semi-structured business documents like invoices and receipts. Unlike plain text, these documents rely heavily on spatial context. The USTC-iFLYTEK team addressed two primary tasks: Key Information Localization and Extraction (KILE) and Line Item Recognition (LIR). Their solution leverages a domain-specific multimodal graph model, GraphDoc, and a novel learnable Merger module.

GraphDoc: Multimodal Graph Modeling

The core of the approach is GraphDoc, a model designed to capture the spatial "grammar" of document layouts. Rather than processing individual words, GraphDoc treats semantic regions (text boxes) as nodes in a graph. This reduces computational complexity and allows the model to focus on the relationships between layout elements.

The GraphDoc architecture utilizes visual, layout, and text embeddings within a graph attention framework for document understanding.

The model incorporates three primary feature streams:

  1. Text Embedding: Utilizing Sentence BERT to encode the semantic content of text boxes.
  2. Layout Embedding: Capturing 2D coordinates and spatial relationships.
  3. Visual Embedding: Using a Swin Transformer and Feature Pyramid Network (FPN) to extract visual cues from document images.

To overcome data scarcity, the team performed massive-scale pre-training on 2.7 million unlabeled document pages using a Masked Sentence Modeling (MSM) task. This allowed the model to learn domain-specific vocabulary and layout patterns before fine-tuning on the competition's annotated data.

The Learnable Merger Module

A standout contribution of this work is the replacement of traditional rule-based grouping with a learnable Merger module. In document processing, multiple text boxes often belong to a single semantic "instance" (e.g., a multi-line address) or a "line item" (a row in a table).

Visualization of the learnable Merger module, showing how attention matrices facilitate instance and line-item grouping.

The Merger module operates in two stages:

  • Instance Merge: It calculates attention scores between text boxes of the same category to determine if they should be grouped into a single instance.
  • Line Item Merge: It groups these instances into coherent table rows for the LIR task.

By learning these groupings through attention mechanisms, the system can adapt to the diverse and complex layouts found in various invoice styles more effectively than rigid, hard-coded geometric rules.

Multi-Scale OCR and Post-processing

To ensure high recall, the team implemented a multi-scale OCR strategy, scaling images by factors of 1.25x, 1.5x, and 1.75x. This approach significantly improved the detection of small text, raising the instance detection recall rate from 0.979 to 0.995. Final predictions were refined through model ensembling and "common sense" geometric post-processing to correct edge-case merging errors where the model mistakenly grouped boxes across large gaps.

Results and Conclusion

The USTC-iFLYTEK approach achieved top-tier performance on the DocILE benchmark. For the KILE task, the model reached an Average Precision (AP) of 71.25% and an F1-score of 74.25%. In the LIR task, it achieved a micro-F1 score of 75.93%. These results underscore the effectiveness of domain-specific pre-training and learnable aggregation modules in handling the variability of business documents. Future work will focus on further automating post-processing to eliminate the need for manual geometric thresholds.

Summary updated

BibTeX
@inproceedings{wang2023ustc,
  author = {Yan Wang and Jun Du and Jiefeng Ma and Pengfei Hu and Zhenrong Zhang and Jianshu Zhang},
  title = {{USTC-iFLYTEK at DocILE: A Multi-modal Approach Using Domain-specific GraphDoc}},
  booktitle = {CLEF (Working Notes)},
  year = {2023},
  url = {https://openreview.net/forum?id=YWGcOzdoi1},
}
Figure