One-shot Text Field labeling using Attention and Belief Propagation for Structure Information Extraction

Mengli Cheng, Minghui Qiu, Xing Shi, Jun Huang, Wei Lin

Introduction

Extracting structured data from documents like receipts, invoices, or forms is a critical task in automated business processes. While Optical Character Recognition (OCR) has matured significantly, the subsequent step of text field labeling—assigning semantic categories such as "Total Price" or "Invoice Number" to recognized text—remains a significant challenge.

Traditional deep learning approaches for this task are notoriously data-hungry, requiring thousands of labeled examples for every new document layout. This lack of scalability is compounded by privacy concerns regarding sensitive documents and the complexity of handling multi-region fields in "crowded" layouts. This paper proposes a novel one-shot learning framework that requires only a single labeled "support" document to accurately extract fields from new "query" documents of the same type.

The process of text field labeling: from raw document image (a) to OCR text (b) and finally semantic labeling (c).

Methodology: The AttnBP Framework

The authors represent documents as sparse graphs where text regions are nodes and spatial relationships are edges. To minimize computational overhead, they utilize a visibility-based graph construction that reduces memory consumption by over 66%. The core architecture, AttnBP, leverages three distinct modules to transfer layout knowledge:

  1. Landmark-to-Field Attention (LFAttn): This module identifies "landmarks"—static text elements like "Date:" or "Total"—and uses an attention mechanism to map the spatial relationships between these anchors and target fields from the support document onto the query document.
  2. Field-to-Field Attention (FFAttn): This captures second-order spatial dependencies between fields themselves (e.g., the relative position of an "item_price" to an "item_name"), providing additional geometric constraints.
  3. Attention-based Belief Propagation (AttnBP): To resolve ambiguities in dense areas or handle fields spanning multiple lines, the model employs a Conditional Random Field (CRF) implemented as a differentiable Belief Propagation module. This allows the network to refine individual labels based on a collective "consensus" of neighboring regions.
One-shot transfer mechanism between a support document and a test document using Landmark-to-Field (LF) and Field-to-Field (FF) edge attention.

Experimental Evaluation

The model was evaluated on two key datasets: DocLL-oneshot, a challenging collection of 1,307 smartphone-captured documents with 3D distortions and uneven lighting, and SROIE-oneshot, an adaptation of the standard scanned receipt dataset.

The results demonstrate the superiority of the AttnBP approach:

  • High Accuracy: The model achieved 96.4% accuracy on DocLL-oneshot and 98.5% on SROIE-oneshot, significantly outperforming rule-based and standard few-shot baselines.
  • Robustness: Unlike rule-based systems, which saw a ~10% performance drop in the presence of background noise, AttnBP remained stable.
  • Scalability: Performance improved further to 97.8% when the number of support examples was increased to five (5-shot learning).

Conclusion

By prioritizing stable geometric relationships over volatile text content and employing a robust graph-based attention mechanism, the AttnBP framework enables efficient, one-shot information extraction. While the system currently relies on consistent "Manhattan layouts" and accurate landmark detection via OCR, it represents a significant step toward scalable, privacy-preserving document intelligence. Future work will explore zero-shot capabilities and increased robustness against extreme physical distortions.

Summary updated

BibTeX
@inproceedings{cheng2020one,
  author = {Mengli Cheng and Minghui Qiu and Xing Shi and Jun Huang and Wei Lin},
  title = {{One-shot Text Field labeling using Attention and Belief Propagation for Structure Information Extraction}},
  booktitle = {ACM Multimedia},
  year = {2020},
  url = {https://openreview.net/forum?id=HLSeVLWBKSd},
}
Figure