Separating Retention from Extraction in the Evaluation of End-to-end Relation Extraction

Bruno Taillé, Vincent Guigue, Geoffrey Scoutheeten, Patrick Gallinari

Abstract

State-of-the-art NLP models can adopt shallow heuristics that limit their generalization capability (McCoy et al., 2019). Such heuristics include lexical overlap with the training set in Named-Entity Recognition (Taillé et al., 2020) and Event or Type heuristics in Relation Extraction (Rosenman et al., 2020). In the more realistic end-to-end RE setting, we can expect yet another heuristic: the mere retention of training relation triples. In this paper, we propose several experiments confirming that retention of known facts is a key factor of performance on standard benchmarks. Furthermore, one experiment suggests that a pipeline model able to use intermediate type representations is less prone to over-rely on retention.

Introduction

The field of Information Extraction, particularly Named Entity Recognition (NER) and Relation Extraction (RE), has seen significant performance gains driven by large pre-trained language models. However, standard evaluation metrics like F1 score may not accurately reflect a model's ability to extract information from novel contexts. We hypothesize that models often rely on a "retention heuristic"—memorizing specific (head, predicate, tail) triples seen during training and regurgitating them when those entities appear in the test set, regardless of the surrounding text. This paper investigates the extent of this bias and how it varies across different model architectures and datasets.

Methods

To investigate the retention heuristic, we evaluate three recent end-to-end RE models: PURE, SpERT, and TABTO. Our methodology consists of two primary strategies:

Lexical Overlap Partitioning

We partition standard test sets into three categories based on their overlap with the training data:

  • Exact Match: The specific (head, predicate, tail) triple was present in the training set.
  • Partial Match: One argument appears in the same position for a training relation of the same type.
  • New: The relation is entirely unseen in the training data.

We also implement a naive "Retention Heuristic" baseline that predicts labels based on the majority training label for a given entity or relation.

Entity Swapping

To test if models truly process context, we identify test sentences with asymmetric relations between entities of the same type (e.g., "Person A killed Person B"). We then swap the entities in the text to create a counter-factual sentence ("Person B killed Person A"). If a model continues to predict the original relation, it indicates a failure to extract information from the context, relying instead on memorized entity associations.

Example of the entity swapping experiment used to probe context sensitivity.

Datasets

We utilize three standard benchmarks with varying degrees of lexical overlap:

  • CoNLL04: High label consistency and many proper nouns, making it highly susceptible to retention.
  • ACE05: Contains more common nouns and pronouns, leading to lower consistency and more partial matches.
  • SciERC: Scientific text with almost zero exact overlap, serving as a control for the retention heuristic.

Results

Our experiments reveal a significant performance gap between seen and unseen relations across all models and datasets.

Performance of RE models across different datasets, partitioned by lexical overlap.

On CoNLL04, models achieve over 90% F1 on "Exact Match" relations, but this drops to approximately 60% for "New" relations. The naive Retention Heuristic baseline achieves a surprisingly high 35.5% F1 on CoNLL04, highlighting how much overlap exists in the benchmark.

The swapping experiment highlights a critical architectural difference. Joint models (SpERT and TABTO) predicted the original memorized relation nearly 60% of the time when entities were swapped, effectively ignoring the text. In contrast, the pipeline model (PURE) made this error only 14.3% of the time. This suggests that PURE's use of intermediate entity type representations, rather than shared lexical features, forces the relation classifier to rely more on context.

Discussion and Conclusion

Our findings demonstrate that current RE benchmarks are heavily influenced by data retention, which can mask a model's actual extraction capabilities. While pipeline architectures show greater robustness to lexical bias, all models exhibit a significant performance decay on novel relations. Future research should focus on developing benchmarks that explicitly control for lexical overlap and further investigate the role of pre-trained knowledge in the retention phenomenon.

Summary updated

Abstract

State-of-the-art NLP models can adopt shallow heuristics that limit their generalization capability (McCoy et al., 2019). Such heuristics include lexical overlap with the training set in Named-Entity Recognition (Taillé et al., 2020) and Event or Type heuristics in Relation Extraction (Rosenman et al., 2020). In the more realistic end-to-end RE setting, we can expect yet another heuristic: the mere retention of training relation triples. In this paper, we propose several experiments confirming that retention of known facts is a key factor of performance on standard benchmarks. Furthermore, one experiment suggests that a pipeline model able to use intermediate type representations is less prone to over-rely on retention.
BibTeX
@inproceedings{taille2021separating,
  author = {Bruno Taillé and Vincent Guigue and Geoffrey Scoutheeten and Patrick Gallinari},
  title = {{Separating Retention from Extraction in the Evaluation of End-to-end Relation Extraction}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=pZkae6VP5_q},
}
Figure