Abstract: Document-level information extraction (doc-IE) plays a pivotal role in the realm of natural language processing (NLP). This paper embarks on a comprehensive review and discussion of contemporary literature related to doc-IE. In addition, we conduct a thorough error analysis using state-of-the-art algorithms, shedding light on their limitations and remaining challenges for tackling the task of doc-IE. Our findings demonstrate that issues like entity coreference resolution and the lack of robust reasoning significantly hinder the effectiveness of document-level information extraction (doc-IE). Additionally, we uncover new challenges, including labeling noise and relation transitivity. The overarching objective of this survey paper is to provide valuable insights that can empower NLP researchers to further advance the performance of doc-IE.
The Shift to Document-Level Extraction
Traditional Information Extraction (IE) has largely focused on the sentence level—identifying entities and their relationships within a single string of text. However, in the real world, information is rarely so tidy. Key details about an event or a relationship are often scattered across multiple sentences or even entire documents.
Extracting this "document-level" information introduces three major hurdles:
- Long-span Context: Models must maintain "memory" over hundreds or thousands of words.
- Entity Coreference: A model needs to realize that "Apple," "the company," and "it" all refer to the same entity to connect the dots.
- Complex Reasoning: Understanding a relationship might require "multi-hop" logic.
Categorizing doc-IE Methodologies
The survey categorizes doc-IE methods into a clear typology based on model architecture:
- Multi-granularity-based: These models process text hierarchically (sentence → paragraph → document) to capture both local details and global context.
- Graph-based: These represent the document as a network where words, entities, or sentences are nodes, and edges represent dependencies or co-occurrence. This is a dominant approach for modeling long-distance relationships.
- Generation-based: Utilizing models like BART, these treat extraction as a sequence-to-sequence task, "writing" out the extracted information in a structured format.
- LLM-based: The newest frontier, using models like GPT-4 or Llama-2 with specialized prompting (e.g., "Link-of-Analogy") to extract data without extensive fine-tuning.
- Task-specific & Evidence-based: Models designed specifically to find "evidence sentences" before performing the final extraction.
Benchmarking Progress
The paper provides a comprehensive list of benchmarks across various domains. For general news, DocRED is the most popular for Relation Extraction, while WikiEvents and RAMS lead for Event Extraction. In the financial sector, ChFinAnn and DocEE provide massive datasets for official disclosures. Biomedical research relies on DGM, CDR, and GDA for chemical and gene interactions. Newer benchmarks like HacRED specifically target "hard cases" involving long-distance arguments.
Evaluation of state-of-the-art (SOTA) models shows that on the DocRED dataset, the best models (like KD-Rb-l) achieve F1 scores around 67.28. In the financial domain, ReDEE achieved a higher F1 of 81.90, suggesting that domain-specific models perform well when document structures are more predictable.
Uncovering the Bottlenecks: A Deep-Dive Error Analysis
By conducting a deep-dive error analysis of state-of-the-art models, the survey reveals several critical bottlenecks that hinder performance:
- The "Coreference" Bottleneck: Entity Coreference Resolution (ECR) remains a massive pain point. If a model fails to link mentions of the same entity, it cannot aggregate information across the document.
- Labeling Noise is Underestimated: In datasets created via "distant supervision," up to 25% of errors in some models are actually due to noisy or missing labels in the training data.
- Reasoning vs. Pattern Matching: Many models rely on surface-level patterns. When a relationship requires logical deduction across different sentences, performance drops sharply.
- Relation Transitivity: Models often miss "implied" facts that require transitive logic (e.g., if A is part of B and B is part of C, then A is part of C).
Future Directions
To move the field forward, the authors identify several critical gaps. Future models should treat coreference resolution as a core component rather than an afterthought and explicitly model logical reasoning paths rather than relying solely on attention mechanisms. There is also a need for better "denoising" techniques for training on automatically labeled data. Finally, while current research treats Relation Extraction and Event Extraction as separate silos, the authors suggest they should be trained together to leverage complementary information.
Summary updated