Language, OCR, Form Independent (LOFI) pipeline for Industrial Document Information Extraction

Chang Oh Yoon, Wonbeen Lee, Seokhwan Jang, Kyuwon Choi, Minsung Jung, Daewoo ChoiEMNLP 2024

TL;DR

This paper introduces LOFI, a novel pipeline for extracting information from industrial documents, particularly those in low-resource languages. LOFI tackles the challenges of language, OCR, and document format variations, achieving high accuracy without requiring extensive pre-training on specific languages.

What Problem Does This Paper Address?

Extracting structured information from visually rich documents (VRDs) is crucial for automating workflows in various industries, such as insurance and finance. However, existing methods often struggle with documents in low-resource languages (LRLs) due to a lack of publicly available datasets and pre-trained models. Furthermore, Optical Character Recognition (OCR) outputs, typically at the word level, often require complex post-processing to identify semantic entities. Finally, the diverse and often custom formats of industrial documents, coupled with image distortions like rotation and noise, pose significant challenges for consistent information extraction. This paper aims to address these three interconnected problems: language independence, OCR independence, and form independence.

Methods

The LOFI pipeline is built upon three core components designed to tackle the identified challenges:

  • Language Independence: LOFI leverages the LiLT (Language-Independent Layout Transformer) model as its backbone. LiLT's architecture allows for the replacement of text encoder layers with language-specific pre-trained language models (PLMs). This enables LOFI to adapt to different languages without needing to retrain the entire model from scratch. The paper highlights the advantage of using language-specific PLMs for more contextually relevant tokenization compared to multilingual models.
  • OCR Independence: To overcome the limitations of varying OCR engine outputs (e.g., character-level, word-level, line-level bounding boxes), LOFI introduces a "Token-level box split" algorithm. This algorithm normalizes bounding box information to a consistent token-level representation, making the downstream SER model independent of the OCR engine's output format. This ensures that the model's performance is not degraded by differences in how OCR engines segment text.
  • Form Independence: For handling diverse document formats and potential distortions, LOFI employs the SPADE decoder. This decoder is designed to operate robustly even with inconsistent order information in the input, effectively learning how tokens within the same semantic entity are connected. This allows the pipeline to extract information accurately regardless of document rotation, distortion, or complex layouts.

Datasets

The paper evaluates LOFI on the following datasets:

  • LRL Business Documents:
    • Korean medical bills (custom dataset)
    • Japanese receipts (custom dataset)
  • Open Datasets:
    • FUNSD (Form Understanding in Noisy Scanned Documents)
    • CORD (Consolidated Receipt Dataset)

The custom datasets for Korean medical bills and Japanese receipts are noted as not publicly available due to personal information security concerns.

Key Insights and Ideas

  • Unified Approach for LRLs: The most significant insight is the development of a single pipeline (LOFI) that effectively addresses language, OCR, and form independence simultaneously, which is crucial for industrial applications in low-resource settings.
  • Token-level Box Splitting is Key: The "Token-level box split" algorithm is a critical innovation. It decouples the information extraction process from the specific output format of OCR engines, a common bottleneck in document understanding.
  • Leveraging Existing Models Wisely: By building upon LiLT and adapting its architecture with language-specific PLMs and the SPADE decoder, the paper demonstrates how to achieve high performance without extensive, language-specific pre-training.
  • Practical Impact: The successful application of LOFI in real-world scenarios like insurance claims and tax-free declaration services underscores its practical value and readiness for industrial deployment.

Key Results

LOFI demonstrated strong performance on both LRL and open datasets:

  • LRL Business Documents:
    • Korean medical bills: Achieved an entity-level F1 score of 95.64% with LOFI-ko.
  • Japanese receipts: Achieved an entity-level F1 score of 94.60% with a multilingual LOFI model.
  • LOFI-ko uses significantly fewer parameters (116M) compared to LayoutXLM (369M), offering advantages in resource utilization and processing speed.
  • Open Datasets (English):
    • FUNSD: LOFI-en achieved an F1 score of 78.99%.
  • CORD: LOFI-en achieved an F1 score of 96.39%.
  • The results on open datasets highlight that LOFI's performance is competitive, though it shows a greater advantage when fine-tuning data is abundant, as seen with CORD compared to FUNSD.

Limitations and Future Work

  • Limitations:
    • Data Dependency: The practical implementation of LOFI is constrained by the need for extensive training data, especially for complex annotation tasks requiring domain expertise.
  • OCR Error Susceptibility: The encoder-based model is susceptible to OCR errors stemming from low-quality images or noise, as it relies directly on OCR output.
  • Reproducibility: The experimental datasets contain sensitive personal information and thus cannot be reproduced.
  • Future Work:
    • Data Augmentation: Developing methods for data augmentation to reduce the need for extensive manual annotation.
  • Efficient Annotation: Exploring more efficient annotation strategies.
  • Improved Decoder Architectures: Investigating advanced decoder architectures to better handle document challenges.
  • Reducing OCR Error Impact: Developing methods to decrease the impact of OCR errors and improve post-processing for enhanced robustness.

Summary updated

BibTeX
@inproceedings{yoon2024language,
  author = {Chang Oh Yoon and Wonbeen Lee and Seokhwan Jang and Kyuwon Choi and Minsung Jung and Daewoo Choi},
  title = {{Language, OCR, Form Independent (LOFI) pipeline for Industrial Document Information Extraction}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
}
Figure