Automatic Error Analysis for Document-level Information Extraction

Aliva Das, Xinya Du, Barry Wang, Kejian Shi, Jiayuan Gu, Thomas Porter, Claire Cardie

Introduction

Standard metrics like Precision, Recall, and F1 are the primary tools for NLP evaluation, yet they often function as "black boxes." They indicate that a model has failed without explaining how or why. This research introduces an automated framework designed to categorize errors in document-level Information Extraction (IE). The findings reveal a compelling narrative: while modern neural models excel at identifying exact text spans, they often struggle with the complex logic required to link entities across an entire document—a task where some 30-year-old systems still hold a competitive edge.

The Challenge of Document-Level IE

Information Extraction has transitioned from simple relation identification within single sentences to complex "template filling" at the document level. For instance, when processing a news report about a disease outbreak, a model must extract the disease name, the affected country, and the victim count, even if these details are scattered across multiple paragraphs or mixed with information about a different event.

The primary difficulty lies in evaluation. If a model fails to extract a victim count, is the error due to a failure in detection or a misassignment to the wrong event? Current evaluation methods provide only a single aggregate score, offering little guidance for model refinement. This work addresses the need for fine-grained, systematic error characterization in document-level IE.

A Framework for Systematic Error Characterization

The authors propose a transformation-based framework inspired by edit distance, applied to complex templates. The process involves three key stages:

  1. Optimized Matching: Predicted templates are recursively matched to "gold" (ground truth) templates to find the mapping that maximizes the overall F1 score.
  2. Transformations: The system calculates the specific "edits" required to transform a prediction into the gold standard. These include altering spans, changing roles (e.g., moving an entity from 'Victim' to 'Perpetrator'), and handling spurious or missing information.
  3. Error Mapping: These transformations are mapped to 13 specific IE error types, categorized into "Within Template" errors, "Cross Template" errors, and "Template Detection" failures.
Overview of the automatic error analysis framework, showing how an input document is processed into predicted and desired templates to generate error statistics.

Benchmarking Across Eras

The framework was used to evaluate models across three distinct domains:

  • MUC-4 (1992): Terrorist incident reports in Latin America.
  • ProMED: Global disease outbreak reports.
  • SciREX: Scientific papers requiring complex 4-ary relation extraction.

A surprising result emerged from the MUC-4 dataset: the GE NLToolset, a system from 1992, achieved an F1 score of 54.29%, outperforming modern neural models like GTT (49.00%) and DyGIE++ (45.79%). The older system's success was driven by its superior recall (52.09% compared to ~36-40% for neural models), indicating that it was better at capturing the full scope of events described in the text.

Key Insights: Span vs. Role Trade-offs

The analysis highlights a clear trade-off in model performance. Modern neural models are significantly better at "Span Extraction"—identifying the precise boundaries of names or entities. They rarely make the boundary mistakes common in older systems, such as including unnecessary context.

However, the 1990s-era systems were more adept at "Role Assignment"—understanding the logical connections between entities. Across all datasets, "Missing Role Fillers" and "Missing Templates" remain the most significant error sources for modern models. These models tend to be overly conservative, often failing to identify events that are mentioned subtly or across long distances in the text. Furthermore, scientific text (SciREX) proved to be the most challenging domain, likely due to document length and specialized vocabulary.

Limitations and Future Directions

While the framework provides deep insights, the matching algorithm is computationally intensive. As the number of templates and role fillers grows, the time required for optimal mapping increases significantly.

Future research will focus on:

  • Improving the efficiency of the matching algorithm, potentially through bipartite matching.
  • Extending the tool to analyze "set-fill" roles with fixed options.
  • Investigating the linguistic factors that cause neural models to consistently miss specific role fillers.

By moving beyond aggregate scores, this automated error analysis framework provides a necessary roadmap for developing the next generation of document-level IE systems.

Summary updated

BibTeX
@inproceedings{das2022automatic,
  author = {Aliva Das and Xinya Du and Barry Wang and Kejian Shi and Jiayuan Gu and Thomas Porter and Claire Cardie},
  title = {{Automatic Error Analysis for Document-level Information Extraction}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.acl-long.274},
}
Figure