Separating Retention from Extraction in the Evaluation of End-to-end Relation Extraction

Bruno Taillé, Vincent Guigue, Geoffrey Scoutheeten, Patrick Gallinari

Introduction

Current evaluation metrics for end-to-end Relation Extraction (RE) often fail to distinguish between a model's ability to understand linguistic context and its ability to memorize facts encountered during training. This paper investigates whether state-of-the-art models are truly "extracting" information from context or merely "retaining" triples. If a model predicts a "Kill" relation between "Lee Harvey Oswald" and "Kennedy" simply because it saw that pair in training, it is performing a lookup rather than contextual extraction. This distinction is crucial for understanding how well models generalize to novel facts in real-world applications.

Methodology

The researchers propose two primary experimental strategies to disentangle retention from extraction:

  1. Lexical Overlap Partitioning: The test set is divided based on its overlap with training data. Entities are categorized as Seen or Unseen, and relations are classified as Exact Match (the full triple was in training), Partial Match (one argument seen in that role), or New.
  2. The Swapping Experiment: To test contextual reliance, the authors identify asymmetric relations (e.g., "A is located in B"). They manually swap the head and tail entities in the input sentence (e.g., "B is located in A"). A model relying on context should predict the new, swapped relation, while a model relying on retention will likely predict the original training triple.

The study evaluates three architectures: PURE (a pipeline model), and SpERT and TABTO (joint models), across the CoNLL04, ACE05, and SciERC datasets.

Results: The Retention Bias

The evaluation reveals a significant performance gap between memorized facts and new information. Across all datasets, F1 scores for "Exact Match" relations—those previously seen in training—are substantially higher than for "New" relations.

Performance comparison across ACE05, CoNLL04, and SciERC datasets, partitioned by lexical overlap.

On the ACE05 dataset, which features high lexical overlap, the performance drop is particularly stark. Models achieve over 80% F1 on exact matches but struggle to reach 45% on entirely new relations. This suggests that a significant portion of the reported performance in literature may be attributed to simple memorization. This is further evidenced by the "naive Retention Heuristic." By simply looking up the most frequent relation for a given entity pair in the training data, this heuristic achieves a 90.9 F1 on CoNLL04's exact matches, nearly matching the performance of complex neural architectures like TABTO (92.6 F1).

Analysis: Joint vs. Pipeline Models

The study also reveals that architectural choices significantly impact a model's reliance on retention. Joint models, which optimize entity and relation extraction simultaneously, tend to be more prone to memorization and "forgetful" of context.

Performance on the 'Located in' relation, illustrating the impact of entity swapping.

In the swapping experiment, the pipeline model PURE demonstrated superior robustness. While joint models like TABTO often predicted the original training triple (the "reverse RE" error) when arguments were swapped, PURE maintained a significantly higher F1 score on the swapped context. For example, in the "Kill" relation on CoNLL04, TABTO's performance plummeted from 89.6 F1 to 27.6 F1, while its reverse RE score was nearly 60%. PURE is likely more robust because it uses predicted entity types (e.g., "Person") as input for the relation classifier rather than raw lexical tokens, forcing the model to attend to linguistic triggers rather than memorized entity pairs.

Conclusion

The findings suggest that current RE benchmarks are heavily influenced by a model's ability to memorize training data. While joint models often achieve high holistic scores, they are more susceptible to retention bias. Pipeline models, by decoupling entity and relation steps, demonstrate better generalization to novel contexts. Future work should focus on creating benchmarks that specifically probe for context-based extraction to ensure models are truly learning to "read" rather than just "remember."

Summary updated

Abstract

State-of-the-art NLP models can adopt shallow heuristics that limit their generalization capability (McCoy et al., 2019). Such heuristics include lexical overlap with the training set in Named-Entity Recognition (Taillé et al., 2020) and Event or Type heuristics in Relation Extraction (Rosenman et al., 2020). In the more realistic end-to-end RE setting, we can expect yet another heuristic: the mere retention of training relation triples. In this paper, we propose several experiments confirming that retention of known facts is a key factor of performance on standard benchmarks. Furthermore, one experiment suggests that a pipeline model able to use intermediate type representations is less prone to over-rely on retention.
BibTeX
@inproceedings{taille2021separating,
  author = {Bruno Taillé and Vincent Guigue and Geoffrey Scoutheeten and Patrick Gallinari},
  title = {{Separating Retention from Extraction in the Evaluation of End-to-end Relation Extraction}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=pZkae6VP5_q},
}
Figure