Extracting Zero-shot Structured Information from Form-like Documents: Pretraining with Keys and Triggers

Rongyu Cao, Ping Luo

Introduction

Extracting structured data from documents like invoices and receipts—a task known as Key Information Extraction (KIE)—is a cornerstone of automated accounting and document processing. However, current state-of-the-art models often treat keys (e.g., "Total Amount") as fixed category labels. This approach leads to two major issues: "Zero-shot Failure," where models cannot find fields not present in the training set, and a "Lack of Explainability," as models rely on global layout patterns rather than the specific semantic cues humans use to locate information.

The KATA Architecture

To address these limitations, we introduce KATA (Key-Aware and Trigger-Aware), a two-stage architecture designed to mimic human search patterns. Unlike traditional models, KATA treats keys as sequences of words rather than atomic IDs, enabling the model to leverage semantic meaning to understand what it is looking for.

The model operates through a two-stage mapping process:

  1. Key-to-Trigger: The model learns to map an input key (e.g., "telephone") to a specific "trigger" phrase found within the document (e.g., "Tel" or "Phone No.").
  2. Trigger-to-Value: Using the identified trigger and the original key, the model locates the actual value, often relying on structural invariants like tabular alignment (e.g., the value is typically to the right of or below the trigger).
Visual representation of the Key-Trigger-Value mapping across different document layouts.

To train this system at scale, the authors automatically constructed a massive dataset using Wikipedia Infoboxes. These infoboxes were re-rendered as PDF documents to simulate diverse form layouts, creating a rich training ground for key-trigger-value relationships with over 1.2 million key-value pairs.

Key Insights

A central thesis of this work is that while document layouts vary, the relationship between a trigger and its value remains relatively constant. Furthermore, the linguistic diversity of triggers is significant; a single key like "telephone" can be represented by dozens of different trigger expressions across various documents.

Proportion of different trigger expressions for a single key, highlighting the complexity of the extraction task.

By encoding keys as text, KATA generalizes effectively. If the model understands how to find a "Telephone," it can likely locate a "Mobile Number" due to their semantic proximity. Ablation studies confirmed that KATA relies heavily on these triggers, whereas standard models like LayoutLM often ignore these critical anchors, relying instead on global layout patterns that fail to generalize to new document types.

Experimental Results

KATA was evaluated on the public SROIE dataset (receipts) and a private Chinese bank statement dataset (Grater). The most striking result was KATA's performance on zero-shot keys—fields entirely absent from the training data. While traditional models like BERT and LayoutLM achieved effectively 0% accuracy on these unseen fields because they rely on fixed labels, KATA-wiki achieved an F1 score of 70.7% on SROIE and 73.4% on Grater.

On standard (non-zero-shot) tasks, KATA remained highly competitive, slightly outperforming LayoutLM on the SROIE dataset (94.7% vs 94.6% F1), proving that adding zero-shot capabilities does not compromise general performance.

Conclusion and Future Work

While KATA demonstrates a breakthrough in zero-shot KIE, limitations remain regarding layout diversity beyond simple left-to-right structures and the lack of value-type constraints (e.g., ensuring a "Date" field contains numeric date patterns). Future research will focus on incorporating more complex document geometries, such as those found in Wikitables, to further enhance the model's robustness against complex real-world forms.

Summary updated

Abstract

In this paper, we revisit the problem of extracting the values of a given set of key fields from form-like documents. It is the vital step to support many downstream applications, such as knowledge base construction, question answering, document comprehension and so on. Previous studies ignore the semantics of the given keys by considering them only as the class labels, and thus might be incapable to handle zero-shot keys. Meanwhile, although these models often leverage the attention mechanism, the learned features might not reflect the true proxy of explanations on why humans would recognize the value for the key, and thus could not well generalize to new documents. To address these issues, we propose a Key-Aware and Trigger-Aware (KATA) extraction model. With the input key, it explicitly learns two mappings, namely from key representations to trigger representations and then from trigger representations to values. These two mappings might be intrinsic and invariant across different keys and documents. With a large training set automatically constructed based on the Wikipedia data, we pre-train these two mappings. Experiments with the fine-tuning step to two applications show that the proposed model achieves more than 70% accuracy for the extraction of zero-shot keys while previous methods all fail.
BibTeX
@inproceedings{cao2021extracting,
  author = {Rongyu Cao and Ping Luo},
  title = {{Extracting Zero-shot Structured Information from Form-like Documents: Pretraining with Keys and Triggers}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i14.17494},
}
Figure