Learning Label Dependencies for Visual Information Extraction

Minghong Yao, Liansheng Zhuang, Houqiang Li, Jiuchang Wei

Abstract

Visual Information Extraction (VIE) aims to extract structured information from visually rich document images. Previous methods often treat VIE as a sequence labeling problem, ignoring inherent label correlations which can degrade performance. This paper proposes a novel framework to exploit label dependencies using a pre-trained Label Transformer to regularize label sequences based on document layout patterns. An Inference Transformer then predicts sequences by considering both entity features and label sequence likelihood. This framework integrates with existing models like LayoutLM and GeoLayoutLM, showing significant performance gains across public datasets.

Introduction: The "Grammar" of Documents

Visual Information Extraction (VIE) is critical for converting documents like receipts and forms into structured data. A primary challenge in Semantic Entity Recognition (SER) is the serialization of 2D documents into 1D sequences. This process often breaks spatial relationships, placing physically adjacent items far apart in the text stream.

While current models excel at aligning text, image, and layout features, they often overlook label-level dependencies. For instance, in a receipt, a "menu" label is typically followed by specific "sub-menu" labels. Traditional sequence models like Conditional Random Fields (CRFs) struggle with these long-range dependencies. This research introduces a framework that learns the "syntax" of document labels to improve extraction accuracy.

Methodology

The proposed framework utilizes Energy-Based Modeling to regularize predicted label sequences through three core components:

  1. Feature Transformer: A standard pre-trained VIE model (e.g., LayoutLMv3) that calculates "feature energy," representing the likelihood of a label based on local text and layout.
  2. Label Transformer: A decoder-only transformer trained to learn the "language" of labels. It assigns lower "label energy" to sequences that follow established document patterns. It employs a causal mask to capture long-distance transitions and utilizes Delta Bbox Embeddings (the coordinate difference between consecutive tokens) to detect layout shifts like line breaks.
  3. Inference Transformer: This component iteratively refines the label sequence. Starting from an initial guess, it performs "denoising" via fixed-point iteration to find the sequence that minimizes the combined Feature and Label Energy.
The proposed framework architecture featuring Feature, Label, and Inference Transformers.

The training employs a bi-level optimization strategy where the Label Transformer learns valid sequences while the Inference Transformer is trained to minimize total energy.

Experimental Evaluation

The framework was tested on three benchmark datasets: FUNSD (complex forms), CORD (Indonesian receipts with 42 subclass labels), and SROIE (receipts focused on key financial fields).

Key Results

The framework consistently improved F1 scores for various backbone architectures:

  • FUNSD: GeoLayoutLM's F1 rose from 92.3 to 93.1, while LayoutLM-large improved significantly from 79.0 to 83.2.
  • CORD: Achieved a state-of-the-art F1 of 98.2 with GeoLayoutLM.
  • SROIE: LayoutLMv2-large reached an F1 of 97.99.
Performance comparison (F1, Precision, Recall) between the original models and the proposed framework across various labels.

Ablation studies highlighted the Label Transformer as the most vital component. Without it, both precision and recall dropped, confirming that label correlations are a crucial missing link in current VIE approaches.

Discussion and Insights

  • Structural Syntax: Document labels follow structural rules analogous to linguistic grammar. Capturing this syntax provides a signal that multimodal features alone might miss.
  • Iterative Refinement: Treating inference as a fixed-point problem allows the model to correct errors where text or image features are ambiguous, leading to more stable predictions.
  • Computational Trade-offs: The iterative nature of the Inference Transformer increases computational overhead during testing. While more iterations generally improve results, performance eventually plateaus.

Conclusion

By explicitly modeling label dependencies, this framework addresses a significant limitation in standard VIE models. The combination of a Label Transformer for structural knowledge and an Inference Transformer for iterative refinement provides a robust mechanism for improving entity recognition in complex documents. Future work may explore extending this approach to other structured prediction tasks where long-range dependencies are prevalent.

Summary updated

Abstract

Visual Information Extraction (VIE), which aims to extract structured information from visually rich document images, has drawn much attention due to its wide applications in document understanding. However, previous methods often treat the VIE task as a sequence labeling problem and ignore the label correlations in the sequence, which may significantly degrade their performance. To address this issue, this paper proposes a novel framework to exploit the potential of label correlations to improve the VIE models' performance. Its key idea is to learn the label dependency of entities, and use it to regularize the label sequence. Specifically, to capture the label dependency of entities, a label transformer is pre-trained to assign a higher likelihood to the label sequence that respects the label patterns of document layouts. During testing stages, an inference transformer is used to predict the label sequence by considering not only the features of each entity but also the likelihood of the label sequence evaluated by the label transformer. Our framework can be combined with existing popular VIE models such as LayoutLM and GeoLayoutLM. Extensive experiments on public datasets have demonstrated the effectiveness of our framework.
BibTeX
@inproceedings{yao2024learning,
  author = {Minghong Yao and Liansheng Zhuang and Houqiang Li and Jiuchang Wei},
  title = {{Learning Label Dependencies for Visual Information Extraction}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=XUQyxvMMbf},
}
Figure