Analysing recall loss in named entity slot filling

Glen Pink, Joel Nothman, James R. CurranEMNLP 2014 · DOI

Abstract

In information extraction, the ability to recover correct answers is fundamentally limited by the initial stages of the processing pipeline. This paper systematically identifies the "leaky" stages of the Slot Filling (SF) pipeline, revealing that while precision can be tuned in later stages, up to 39% of correct answers are discarded by early-stage filters and can never be recovered. We quantify recall loss across document retrieval, candidate generation, and coreference resolution, providing a roadmap for improving the recall of state-of-the-art extraction systems.

Introduction

Slot Filling (SF) is the task of automatically populating a database with attributes (slots) for a given entity—for example, identifying the "spouse" or "employer" of a person mentioned in a massive text corpus. A major bottleneck in current SF systems is recall. Most state-of-the-art systems are "precise but blind"; they employ complex filters to manage the overwhelming volume of data, but these filters often discard correct answers before the system has an opportunity to analyze them. Because a candidate that is never generated cannot be recovered, understanding where this "recall loss" occurs is the first step toward building more robust extraction systems.

Methodology

The researchers treat the standard SF pipeline as a series of increasingly restrictive filters. To measure recall loss at each stage, they implemented a collection of naive SF systems to track how many correct answers survive each processing step.

The standard Slot Filling pipeline architecture, from document retrieval to answer merging and ranking.

The pipeline stages analyzed include:

  1. Document Retrieval: Comparing "Oracle" retrieval (knowing exactly which documents contain the answer) against "Exact Match" (searching for the entity name verbatim).
  2. Candidate Generation: Transitioning from raw noun-phrase (NP) n-grams to Named Entities (NEs) identified by a tagger.
  3. Type Constraints: Filtering candidates based on whether their entity type (e.g., Person, Organization) matches the required slot type.
  4. Sentence & Coreference Filters: Evaluating recall changes when the query and answer must appear in the same sentence, and measuring the impact of coreference resolution (linking pronouns or titles back to the original entity).
  5. Bootstrapping Reachability: A graph-based bootstrapping approach was used to determine the theoretical upper bound of what a system could learn from training data.

Key Insights

The "NER Tax" and Type Constraints

Approximately 10% of recall is lost because Named Entity Recognition (NER) tools fail to tag a word correctly or misidentify name boundaries. However, applying entity type constraints proved to be a highly efficient filter: it reduced the search space by over 80% while incurring only a 2% recall hit.

The Coreference Trade-off

Coreference resolution is a double-edged sword. While it boosts recall by approximately 16%, it also doubles the search space, increasing the potential for false positives. Interestingly, 12% of the 16% recall gained by complex coreference systems can be recovered using simple name-matching rules (e.g., matching "John Kerry" to "Kerry"), suggesting that expensive coreference engines may not always be necessary for certain types of entity matching.

Quantitative Results

The authors quantified the "survival rate" of correct answers as they pass through the pipeline. The cumulative loss demonstrates a "leaky pipe" effect where the final extraction stage is blind to nearly 40% of the correct answers due to hard constraints in candidate generation.

The "leaky pipe" of the SF pipeline, showing the percentage of recall preserved and the reduction in candidate count at each stage.

As shown in the figure above:

  • 97% Recall: Maintained when considering all NP n-grams.
  • 90% Recall: After filtering for Named Entities (7% loss to NER errors).
  • 88% Recall: After applying entity type constraints.
  • 80% Recall: When requiring the query and answer to be in the same sentence using full coreference.
  • 64% Recall: When requiring the query and answer to be in the same sentence without coreference.

In bootstrapping experiments, the maximum reachable recall was found to be only 39% to 43%, limited by the fact that many relations in the corpus are unique or do not follow repeatable patterns.

Conclusion and Future Work

The analysis highlights that significant recall is lost early in the pipeline, particularly during NER and sentence-level filtering. Future work should focus on document-level extraction rather than sentence-level extraction, as 8% of recall is lost simply because queries and fills appear in different sentences. Moving beyond the sentence boundary is essential for the next generation of high-recall systems.

Summary updated

BibTeX
@inproceedings{pink2014analysing,
  author = {Glen Pink and Joel Nothman and James R. Curran},
  title = {{Analysing recall loss in named entity slot filling}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2014},
  doi = {10.3115/v1/d14-1089},
}
Figure