TL;DR
This paper introduces "Kleister," a new benchmark and two datasets designed to push the boundaries of Information Extraction (IE) for long, complex documents. By focusing on real-world scenarios with challenging layouts and requiring deeper contextual understanding, Kleister aims to bridge the gap between current NLP capabilities and practical business needs.
What Problem Does This Paper Address?
Modern NLP models excel at understanding short texts, but they often falter when faced with long, formal documents that incorporate spatial layout, tables, forms, and multi-page structures. Extracting specific information from these documents, like contract details or financial figures, is crucial for businesses but remains a significant challenge. Existing datasets often focus on shorter documents or simpler layouts, leaving a gap in evaluating IE systems on complex, real-world documents that require understanding beyond just the textual content. This paper addresses this gap by introducing a new task and datasets that mimic these real-world complexities.
Methods
The authors propose a "Pipeline" method as a baseline for their IE task. This method involves several stages:
- Auto-tagging: Using regular expressions and gold-standard values to identify potential entities in training data. This is primarily for creating training data for the NER model.
- Named Entity Recognition (NER): Employing models like Flair, BERT, and RoBERTa to detect entities within the text.
- Normalization: Standardizing extracted entities into a canonical form (e.g., dates to YYYY-MM-DD).
- Aggregation: Combining multiple detected candidates for an entity into a single output, typically by selecting the one with the highest confidence score.
A key aspect of their evaluation involves assessing the impact of different PDF processing tools (Tesseract, Textract, pdf2djvu) on the overall IE performance, as the quality of text extraction from these complex documents is critical.
Datasets
The paper introduces two new datasets for the Kleister task:
- Kleister-NDA: Contains US non-disclosure agreements. These are "born-digital" documents.
- Kleister-Charity: Consists of annual financial reports from UK charitable foundations. These documents were often scanned and required OCR.
These datasets are designed to be challenging due to their length, complex layouts (tables, forms, graphics), and the need for document-level context to extract information accurately. They differ from existing datasets like WikiReading by focusing on formal documents with intricate layouts and noisy OCR inputs.
Key Insights and Ideas
- Layout Matters: The spatial arrangement of text and elements within a document is as crucial as the text itself for accurate information extraction, especially in complex layouts.
- OCR Quality is a Bottleneck: The performance of IE systems is highly sensitive to the quality of text extracted from PDFs. Even sophisticated NLP models struggle when the input text is noisy or incomplete due to poor OCR.
- Document-Level Context is Essential: Extracting entities often requires understanding the entire document, not just isolated sentences. This is particularly true for inferring relationships or resolving ambiguities.
- Normalization is Non-Trivial: Standardizing extracted information (like dates, addresses, or monetary values) into a consistent format is a significant challenge, especially when documents present data in diverse ways.
- Business Logic is Key: Real-world IE often requires incorporating domain-specific business rules to disambiguate entities or select the correct information when multiple options exist.
Key Results
- pdf2djvu was the best PDF processing tool for the born-digital Kleister-NDA dataset, achieving an F-score of 77.70% with Flair.
- For the Kleister-Charity dataset (mixed born-digital and scans), Textract performed best, with Flair achieving an F-score of 80.10%.
- The RoBERTa-base model generally outperformed Flair and BERT, especially for entities like monetary amounts and organization names.
- Despite using advanced models, the F-scores for specific entities, particularly those involving complex normalization or business logic (e.g.,
term,address_*), remained significantly lower than the human baseline, highlighting the difficulty of the task. For instance, thetermentity in the NDA dataset had an F-score as low as 40.23% with BERT. - The paper identified several common problems, including normalization issues (e.g., "Ltd" vs. "Limited"), layout complexity, document-level context understanding, and business logic ambiguities.
Limitations and Future Work
- Text-Only Baseline: The current Pipeline method relies solely on textual information and does not leverage visual layout features, which is a significant limitation for documents with complex spatial structures.
- OCR Dependency: The performance is heavily dependent on the quality of PDF text extraction, and errors introduced by OCR can severely impact results.
- Generalization: While the datasets are designed for real-world scenarios, further work is needed to ensure models generalize well across a wider variety of document types and layouts.
- Future Work: The authors suggest exploring methods that incorporate visual features, improving OCR robustness, and developing more sophisticated models that can better handle document-level context and business logic. They also plan to set up a shared-task platform for further evaluation.
Summary updated