Jointly Learning Span Extraction and Sequence Labeling for Information Extraction from Business Documents

Nguyen Hong Son, Hieu M. Vu, Tuan-Anh D. Nguyen, Minh-Tien Nguyen

Introduction

Extracting structured information from business documents—such as legal contracts, bidding notices, or permit licenses—presents unique challenges for standard Natural Language Processing (NLP) models. These documents are typically very long, often spanning 30 to 50 pages, yet the target information is extremely sparse, frequently accounting for less than 1% of the total text.

Furthermore, traditional Machine Reading Comprehension (MRC) models, while precise at identifying single spans, often struggle with fields that appear multiple times throughout a document. Conversely, Named Entity Recognition (NER) models handle multiple spans well but lack the contextual depth required for sparse extraction in long documents. Standard Transformer-based models like BERT-QA also suffer from significant computational overhead because they require re-encoding the entire document for every query. This paper proposes a hybrid Information Extraction (IE) model that jointly learns span extraction and sequence labeling, achieving state-of-the-art accuracy while delivering a 10x speedup over traditional methods.

Methodology: The Two-Channel Architecture

The core innovation of this research is a "two-channel" encoding strategy that decouples document encoding from query processing, treating IE as a multi-task learning problem.

The proposed joint SpanIE and NER architecture featuring two-channel encoding and an aggregation layer.

Two-Channel Encoding

Unlike standard BERT-QA, which concatenates the query and document for a single pass through the encoder, this model encodes the document only once. Queries (the specific fields to be extracted) are processed through a separate, lightweight embedding layer. This allows the model to handle all queries in parallel, reducing the encoding complexity from $O(m)$ to $O(1)$ relative to the number of fields ($m$).

Joint Learning Heads

The model employs two distinct heads that share the document's hidden states:

  1. SpanIE Head: Utilizes a "Multiple General Attention" mechanism to correlate query embeddings with document tokens, predicting the start and end positions of the primary answer.
  2. NER Head: Predicts BIO (Beginning, Inside, Outside) tags for every token. This sequence labeling task provides a denser training signal that helps the model learn better document representations, even for fields that only have a single answer.

Aggregation Algorithm

To produce final predictions, an aggregation algorithm prioritizes the high-precision results from the SpanIE head and supplements them with non-overlapping spans identified by the NER head. This hybrid approach captures both the precision of MRC and the coverage of NER, effectively handling fields that appear multiple times in a single document.

Experimental Evaluation

The model was evaluated on three primary datasets across two languages: License (Japanese permit licenses), Bidding (Japanese business bidding scenarios), and CUAD-small (a subset of the English Contract Understanding Atticus Dataset). These datasets represent a range of document lengths and field complexities.

Key Results and Discussion

The joint SpanIE+NER model consistently outperformed baselines in both accuracy and efficiency across all tested domains.

Performance comparison (F1-score) between SpanIE+NER and BERT-QA across the License, Bidding, and CUAD-small datasets.

Accuracy and Convergence

On the Bidding test set, the model achieved an F1-score of 0.9244, significantly surpassing standard BERT-QA (0.8915) and pure NER (0.7043). The researchers found that the NER head provided "gradient synergy," helping the SpanIE head converge faster and perform better by enriching the document's hidden state representations. This synergy allows the model to reach high performance levels with fewer training iterations compared to single-task methods.

Inference Speedup

The most dramatic improvement was observed in processing time. On the CUAD-small dataset, which contains long legal documents, standard BERT-QA required over 10 minutes to process the test set. In contrast, the proposed SpanIE+NER model completed the same task in just 55 seconds—a speedup of approximately 10x. This efficiency makes the model highly suitable for large-scale industrial applications where processing thousands of long documents is required.

Conclusion

By separating query and document encoding and training on joint extraction tasks, the researchers have created a model that is both more accurate and significantly more efficient for real-world business applications. While the model is currently limited by the underlying Transformer's context window, future work will focus on refining decoding methods for complex, overlapping entities and investigating even more advanced architectures for diverse business domains.

Summary updated

Abstract

This paper introduces a new information extraction model for business documents. Different from prior studies which only base on span extraction or sequence labeling, the model takes into account advantage of both span extraction and sequence labeling. The combination allows the model to deal with long documents with sparse information (the small amount of extracted information). The model is trained end-to-end to jointly optimize the two tasks in a unified manner. Experimental results on four business datasets in English and Japanese show that the model achieves promising results and is significantly faster than the normal span-based extraction method. The code is also available.
BibTeX
@inproceedings{son2022jointly,
  author = {Nguyen Hong Son and Hieu M. Vu and Tuan-Anh D. Nguyen and Minh-Tien Nguyen},
  title = {{Jointly Learning Span Extraction and Sequence Labeling for Information Extraction from Business Documents}},
  booktitle = {IJCNN},
  year = {2022},
  url = {https://openreview.net/forum?id=vwDIA7LsS9},
}
Figure