Abstract
Document-level information extraction (IE) requires models to synthesize information across multiple paragraphs to populate complex templates. While standard metrics like the F1 score quantify overall performance, they offer little insight into specific error modalities. This paper introduces a transformation-based framework that automatically categorizes IE errors into 13 types, such as span inaccuracies, incorrect role assignments, and missing templates. By applying this framework to both modern neural models and classic systems from the 1990s, the authors identify critical bottlenecks in current state-of-the-art approaches, particularly regarding recall and coreference resolution.
Introduction
Information Extraction (IE) has evolved from simple sentence-level relation extraction to sophisticated document-level tasks. In this context, a model must identify entities and their roles across an entire text to fill predefined templates. However, the "black box" nature of aggregate metrics like Precision, Recall, and F1 hinders the diagnostic process. Researchers often cannot discern whether a model is failing due to poor span detection, a failure to resolve coreferences (e.g., linking "the virus" to "Newcastle disease"), or complete hallucinations.
Methodology: Automated Error Analysis
The proposed framework utilizes an "edit distance" approach to categorize errors systematically. The process involves three primary stages:
- Optimized Matching: A recursive algorithm identifies the optimal mapping between predicted templates and "gold standard" templates to maximize the global F1 score.
- Atomic Transformations: The system defines operations required to convert a predicted template into the ground truth. These include
Alter Span(adjusting boundaries),Alter Role(reassigning fillers), andRemove/Introduceoperations. - Error Mapping: These transformations are mapped to 13 specific error types, providing a granular view of model performance.
A key innovation is the Span Comparison Score (SCS), which uses the geometric mean of overlaps to provide a more nuanced measure of "closeness" than binary matching.
Datasets and Experimental Setup
The framework was evaluated on three distinct datasets:
- MUC-4 (1992): News reports on Latin American terrorism, enabling a direct comparison with 30-year-old hand-crafted systems.
- ProMED: Reports on global disease outbreaks.
- SciREX: Long-form scientific articles, presenting significant challenges due to document length (avg. 5,000+ tokens) and technical complexity.
Key Insights
The "Old Guard" vs. Modern Neural Models
Surprisingly, the 1992 GE NLToolset outperformed modern neural models (GTT and DyGIE++) on the MUC-4 dataset, achieving a 54.29% F1 score compared to GTT's 49.00%. While modern models are "smarter" at identifying precise span boundaries (fewer Span Errors), they struggle with the underlying logic of the task. They frequently fail at coreference resolution, leading to Duplicate Role Fillers, and are generally more conservative, resulting in high precision but significantly lower recall.
The Recall Bottleneck
Across all datasets, "Missing Role Fillers" and "Missing Templates" were the most prevalent error types. This suggests that the primary challenge for current IE models is not hallucination, but rather an inability to find and extract a large portion of the relevant information scattered throughout the text.
Scientific Text Challenges
Performance on SciREX was notably lower across all models. The extreme length of scientific documents and the lack of domain-specific pre-training make these texts the "final frontier" for document-level IE.
Conclusion
This automated error analysis framework moves beyond aggregate scores to provide actionable insights into model failures. The findings highlight a regression in template logic and coreference handling in modern neural models compared to their hand-crafted predecessors. Future work will focus on improving computational efficiency for long documents and identifying the linguistic root causes of the categorized errors.
Summary updated