Introduction
The Document Information Localization and Extraction (DocILE) 2023 challenge presents a significant hurdle in Document AI: extracting structured data from semi-structured business documents like invoices and receipts. Unlike plain text, these documents rely heavily on spatial context. The USTC-iFLYTEK team addressed two primary tasks: Key Information Localization and Extraction (KILE) and Line Item Recognition (LIR). Their solution leverages a domain-specific multimodal graph model, GraphDoc, and a novel learnable Merger module.
GraphDoc: Multimodal Graph Modeling
The core of the approach is GraphDoc, a model designed to capture the spatial "grammar" of document layouts. Rather than processing individual words, GraphDoc treats semantic regions (text boxes) as nodes in a graph. This reduces computational complexity and allows the model to focus on the relationships between layout elements.
The model incorporates three primary feature streams:
- Text Embedding: Utilizing Sentence BERT to encode the semantic content of text boxes.
- Layout Embedding: Capturing 2D coordinates and spatial relationships.
- Visual Embedding: Using a Swin Transformer and Feature Pyramid Network (FPN) to extract visual cues from document images.
To overcome data scarcity, the team performed massive-scale pre-training on 2.7 million unlabeled document pages using a Masked Sentence Modeling (MSM) task. This allowed the model to learn domain-specific vocabulary and layout patterns before fine-tuning on the competition's annotated data.
The Learnable Merger Module
A standout contribution of this work is the replacement of traditional rule-based grouping with a learnable Merger module. In document processing, multiple text boxes often belong to a single semantic "instance" (e.g., a multi-line address) or a "line item" (a row in a table).
The Merger module operates in two stages:
- Instance Merge: It calculates attention scores between text boxes of the same category to determine if they should be grouped into a single instance.
- Line Item Merge: It groups these instances into coherent table rows for the LIR task.
By learning these groupings through attention mechanisms, the system can adapt to the diverse and complex layouts found in various invoice styles more effectively than rigid, hard-coded geometric rules.
Multi-Scale OCR and Post-processing
To ensure high recall, the team implemented a multi-scale OCR strategy, scaling images by factors of 1.25x, 1.5x, and 1.75x. This approach significantly improved the detection of small text, raising the instance detection recall rate from 0.979 to 0.995. Final predictions were refined through model ensembling and "common sense" geometric post-processing to correct edge-case merging errors where the model mistakenly grouped boxes across large gaps.
Results and Conclusion
The USTC-iFLYTEK approach achieved top-tier performance on the DocILE benchmark. For the KILE task, the model reached an Average Precision (AP) of 71.25% and an F1-score of 74.25%. In the LIR task, it achieved a micro-F1 score of 75.93%. These results underscore the effectiveness of domain-specific pre-training and learnable aggregation modules in handling the variability of business documents. Future work will focus on further automating post-processing to eliminate the need for manual geometric thresholds.
Summary updated