Learning Label Dependencies for Visual Information Extraction

Minghong Yao, Liansheng Zhuang, Houqiang Li, Jiuchang Wei

Abstract

Visual Information Extraction (VIE) from documents like receipts and forms is often treated as an independent sequence tagging task, which overlooks the structural dependencies between labels. This paper addresses this limitation by proposing a framework that explicitly models label correlations using a dual-transformer architecture. By combining a Feature Transformer for content-based likelihood with a Label Transformer for sequence-based grammar, the model optimizes predictions through an iterative Inference Transformer. This approach, grounded in energy-based regularization, significantly improves performance on benchmark datasets such as CORD, FUNSD, and SROIE, particularly in disambiguating visually similar but contextually distinct entities.

Introduction

Visual Information Extraction (VIE) involves extracting structured data from visually rich documents. While modern models leverage text, image, and layout features, they frequently struggle with ambiguous segments where content alone is insufficient. For instance, in a restaurant receipt, a "menu item" is typically followed by "sub-menu items." Standard models often confuse these because their local features are nearly identical. Furthermore, the serialization of 2D layouts into 1D strings creates long-range dependencies that traditional sequence models fail to capture.

Figure 1: Ambiguity in receipt labeling where text and layout features alone fail to distinguish between menu and sub-menu items.

Methodology

The proposed framework introduces an energy-based approach to VIE, utilizing three specialized neural networks to regularize and refine predictions:

  1. Feature Transformer: Utilizes a pre-trained multimodal backbone (e.g., LayoutLMv3) to extract features and calculate "Feature Energy," representing the likelihood of a label based on document content.
  2. Label Transformer: A decoder-only transformer pre-trained on ground-truth label sequences. It learns the "grammar" of document labels (e.g., the sequence of "Subtotal" followed by "Total") to calculate "Label Energy."
  3. Inference Transformer: Acts as an optimizer that predicts a label sequence minimizing the total energy (Feature + Label Energy).

A critical innovation is the use of Fixed-Point Iteration. The Inference Transformer iteratively refines the label sequence, effectively "denoising" initial predictions by considering the surrounding label context until a stable state is reached.

Figure 2: The dual-transformer architecture featuring Feature, Label, and Inference components for energy-based extraction.

Experimental Evaluation

The framework was evaluated on three standard benchmarks: FUNSD (form understanding), CORD (Indonesian receipts with hierarchical labels), and SROIE (key field extraction).

Statistical analysis confirmed that labels are spatially and sequentially correlated. For example, in the CORD dataset, specific menu-related labels exhibit strong clustering patterns. By treating extraction as an energy-minimization problem, the Label Transformer acts as a "critic," penalizing sequences that violate learned document structures.

Key Results

The framework consistently outperformed state-of-the-art baselines:

  • CORD: Achieved a 98.2 F1 score using GeoLayoutLM as the backbone.
  • FUNSD: Improved LayoutLM-large performance from 82.8 to 86.7 F1.
  • SROIE: Reached near-perfect results, with LayoutLMv2-large hitting 97.99 F1.

Ablation studies highlighted the Label Transformer as the most critical component for capturing fine-grained patterns. While the iterative refinement process adds computational complexity, the authors found that four iterations provide an optimal balance between accuracy and processing time.

Conclusion

By explicitly modeling label dependencies, this framework overcomes the limitations of independent sequence tagging in VIE. The combination of energy-based regularization and iterative refinement allows models to correct mistakes using contextual label patterns, setting new performance standards across diverse document types.

Summary updated

BibTeX
@inproceedings{yao2024learning,
  author = {Minghong Yao and Liansheng Zhuang and Houqiang Li and Jiuchang Wei},
  title = {{Learning Label Dependencies for Visual Information Extraction}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=XUQyxvMMbf},
}
Figure