Kleister: A novel task for Information Extraction involving Long Documents with Complex Layout

Filip Gralinski, Tomasz Stanislawek, Anna Wróblewska, Dawid Lipinski, Agnieszka Kaliska, Paulina Rosalska, Bartosz Topolski, Przemyslaw BiecekarXiv 2020 · OpenReview

TL;DR

This paper introduces "Kleister," a new benchmark and two datasets designed to push the boundaries of Information Extraction (IE) for long, complex documents. By focusing on real-world scenarios with challenging layouts and requiring deeper contextual understanding, Kleister aims to bridge the gap between current NLP capabilities and practical business needs.

What Problem Does This Paper Address?

Modern NLP models excel at understanding short texts, but they often falter when faced with long, formal documents that incorporate spatial layout, tables, forms, and multi-page structures. Extracting specific information from these documents, like contract details or financial figures, is crucial for businesses but remains a significant challenge. Existing datasets often focus on shorter documents or simpler layouts, leaving a gap in evaluating IE systems on complex, real-world documents that require understanding beyond just the textual content. This paper addresses this gap by introducing a new task and datasets that mimic these real-world complexities.

Methods

The authors propose a "Pipeline" method as a baseline for their IE task. This method involves several stages:

  1. Auto-tagging: Using regular expressions and gold-standard values to identify potential entities in training data. This is primarily for creating training data for the NER model.
  2. Named Entity Recognition (NER): Employing models like Flair, BERT, and RoBERTa to detect entities within the text.
  3. Normalization: Standardizing extracted entities into a canonical form (e.g., dates to YYYY-MM-DD).
  4. Aggregation: Combining multiple detected candidates for an entity into a single output, typically by selecting the one with the highest confidence score.

A key aspect of their evaluation involves assessing the impact of different PDF processing tools (Tesseract, Textract, pdf2djvu) on the overall IE performance, as the quality of text extraction from these complex documents is critical.

Datasets

The paper introduces two new datasets for the Kleister task:

  • Kleister-NDA: Contains US non-disclosure agreements. These are "born-digital" documents.
  • Kleister-Charity: Consists of annual financial reports from UK charitable foundations. These documents were often scanned and required OCR.

These datasets are designed to be challenging due to their length, complex layouts (tables, forms, graphics), and the need for document-level context to extract information accurately. They differ from existing datasets like WikiReading by focusing on formal documents with intricate layouts and noisy OCR inputs.

Key Insights and Ideas

  • Layout Matters: The spatial arrangement of text and elements within a document is as crucial as the text itself for accurate information extraction, especially in complex layouts.
  • OCR Quality is a Bottleneck: The performance of IE systems is highly sensitive to the quality of text extracted from PDFs. Even sophisticated NLP models struggle when the input text is noisy or incomplete due to poor OCR.
  • Document-Level Context is Essential: Extracting entities often requires understanding the entire document, not just isolated sentences. This is particularly true for inferring relationships or resolving ambiguities.
  • Normalization is Non-Trivial: Standardizing extracted information (like dates, addresses, or monetary values) into a consistent format is a significant challenge, especially when documents present data in diverse ways.
  • Business Logic is Key: Real-world IE often requires incorporating domain-specific business rules to disambiguate entities or select the correct information when multiple options exist.

Key Results

  • pdf2djvu was the best PDF processing tool for the born-digital Kleister-NDA dataset, achieving an F-score of 77.70% with Flair.
  • For the Kleister-Charity dataset (mixed born-digital and scans), Textract performed best, with Flair achieving an F-score of 80.10%.
  • The RoBERTa-base model generally outperformed Flair and BERT, especially for entities like monetary amounts and organization names.
  • Despite using advanced models, the F-scores for specific entities, particularly those involving complex normalization or business logic (e.g., term, address_*), remained significantly lower than the human baseline, highlighting the difficulty of the task. For instance, the term entity in the NDA dataset had an F-score as low as 40.23% with BERT.
  • The paper identified several common problems, including normalization issues (e.g., "Ltd" vs. "Limited"), layout complexity, document-level context understanding, and business logic ambiguities.

Limitations and Future Work

  • Text-Only Baseline: The current Pipeline method relies solely on textual information and does not leverage visual layout features, which is a significant limitation for documents with complex spatial structures.
  • OCR Dependency: The performance is heavily dependent on the quality of PDF text extraction, and errors introduced by OCR can severely impact results.
  • Generalization: While the datasets are designed for real-world scenarios, further work is needed to ensure models generalize well across a wider variety of document types and layouts.
  • Future Work: The authors suggest exploring methods that incorporate visual features, improving OCR robustness, and developing more sophisticated models that can better handle document-level context and business logic. They also plan to set up a shared-task platform for further evaluation.

Summary updated

Abstract

State-of-the-art solutions for Natural Language Processing (NLP) are able to capture a broad range of contexts, like the sentence-level context or document-level context for short documents. But these solutions are still struggling when it comes to longer, real-world documents with the information encoded in the spatial structure of the document, such as page elements like tables, forms, headers, openings or footers; complex page layout or presence of multiple pages. To encourage progress on deeper and more complex Information Extraction (IE) we introduce a new task (named Kleister) with two new datasets. Utilizing both textual and structural layout features, an NLP system must find the most important information, about various types of entities, in long formal documents. We propose Pipeline method as a text-only baseline with different Named Entity Recognition architectures (Flair, BERT, RoBERTa). Moreover, we checked the most popular PDF processing tools for text extraction (pdf2djvu, Tesseract and Textract) in order to analyze behavior of IE system in presence of errors introduced by these tools.
BibTeX
@misc{gralinski2020kleister,
  author = {Filip Gralinski and Tomasz Stanislawek and Anna Wróblewska and Dawid Lipinski and Agnieszka Kaliska and Paulina Rosalska and Bartosz Topolski and Przemyslaw Biecek},
  title = {{Kleister: A novel task for Information Extraction involving Long Documents with Complex Layout}},
  year = {2020},
  eprint = {2003.02356},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2003.02356},
}
Figure