Abstract
Visual Information Extraction (VIE) aims to extract structured information from visually rich document images. Previous methods often treat VIE as a sequence labeling problem, ignoring inherent label correlations which can degrade performance. This paper proposes a novel framework to exploit label dependencies using a pre-trained Label Transformer to regularize label sequences based on document layout patterns. An Inference Transformer then predicts sequences by considering both entity features and label sequence likelihood. This framework integrates with existing models like LayoutLM and GeoLayoutLM, showing significant performance gains across public datasets.
Introduction: The "Grammar" of Documents
Visual Information Extraction (VIE) is critical for converting documents like receipts and forms into structured data. A primary challenge in Semantic Entity Recognition (SER) is the serialization of 2D documents into 1D sequences. This process often breaks spatial relationships, placing physically adjacent items far apart in the text stream.
While current models excel at aligning text, image, and layout features, they often overlook label-level dependencies. For instance, in a receipt, a "menu" label is typically followed by specific "sub-menu" labels. Traditional sequence models like Conditional Random Fields (CRFs) struggle with these long-range dependencies. This research introduces a framework that learns the "syntax" of document labels to improve extraction accuracy.
Methodology
The proposed framework utilizes Energy-Based Modeling to regularize predicted label sequences through three core components:
- Feature Transformer: A standard pre-trained VIE model (e.g., LayoutLMv3) that calculates "feature energy," representing the likelihood of a label based on local text and layout.
- Label Transformer: A decoder-only transformer trained to learn the "language" of labels. It assigns lower "label energy" to sequences that follow established document patterns. It employs a causal mask to capture long-distance transitions and utilizes Delta Bbox Embeddings (the coordinate difference between consecutive tokens) to detect layout shifts like line breaks.
- Inference Transformer: This component iteratively refines the label sequence. Starting from an initial guess, it performs "denoising" via fixed-point iteration to find the sequence that minimizes the combined Feature and Label Energy.
The training employs a bi-level optimization strategy where the Label Transformer learns valid sequences while the Inference Transformer is trained to minimize total energy.
Experimental Evaluation
The framework was tested on three benchmark datasets: FUNSD (complex forms), CORD (Indonesian receipts with 42 subclass labels), and SROIE (receipts focused on key financial fields).
Key Results
The framework consistently improved F1 scores for various backbone architectures:
- FUNSD: GeoLayoutLM's F1 rose from 92.3 to 93.1, while LayoutLM-large improved significantly from 79.0 to 83.2.
- CORD: Achieved a state-of-the-art F1 of 98.2 with GeoLayoutLM.
- SROIE: LayoutLMv2-large reached an F1 of 97.99.
Ablation studies highlighted the Label Transformer as the most vital component. Without it, both precision and recall dropped, confirming that label correlations are a crucial missing link in current VIE approaches.
Discussion and Insights
- Structural Syntax: Document labels follow structural rules analogous to linguistic grammar. Capturing this syntax provides a signal that multimodal features alone might miss.
- Iterative Refinement: Treating inference as a fixed-point problem allows the model to correct errors where text or image features are ambiguous, leading to more stable predictions.
- Computational Trade-offs: The iterative nature of the Inference Transformer increases computational overhead during testing. While more iterations generally improve results, performance eventually plateaus.
Conclusion
By explicitly modeling label dependencies, this framework addresses a significant limitation in standard VIE models. The combination of a Label Transformer for structural knowledge and an Inference Transformer for iterative refinement provides a robust mechanism for improving entity recognition in complex documents. Future work may explore extending this approach to other structured prediction tasks where long-range dependencies are prevalent.
Summary updated