Introduction
Current evaluation metrics for end-to-end Relation Extraction (RE) often fail to distinguish between a model's ability to understand linguistic context and its ability to memorize facts encountered during training. This paper investigates whether state-of-the-art models are truly "extracting" information from context or merely "retaining" triples. If a model predicts a "Kill" relation between "Lee Harvey Oswald" and "Kennedy" simply because it saw that pair in training, it is performing a lookup rather than contextual extraction. This distinction is crucial for understanding how well models generalize to novel facts in real-world applications.
Methodology
The researchers propose two primary experimental strategies to disentangle retention from extraction:
- Lexical Overlap Partitioning: The test set is divided based on its overlap with training data. Entities are categorized as Seen or Unseen, and relations are classified as Exact Match (the full triple was in training), Partial Match (one argument seen in that role), or New.
- The Swapping Experiment: To test contextual reliance, the authors identify asymmetric relations (e.g., "A is located in B"). They manually swap the head and tail entities in the input sentence (e.g., "B is located in A"). A model relying on context should predict the new, swapped relation, while a model relying on retention will likely predict the original training triple.
The study evaluates three architectures: PURE (a pipeline model), and SpERT and TABTO (joint models), across the CoNLL04, ACE05, and SciERC datasets.
Results: The Retention Bias
The evaluation reveals a significant performance gap between memorized facts and new information. Across all datasets, F1 scores for "Exact Match" relations—those previously seen in training—are substantially higher than for "New" relations.
On the ACE05 dataset, which features high lexical overlap, the performance drop is particularly stark. Models achieve over 80% F1 on exact matches but struggle to reach 45% on entirely new relations. This suggests that a significant portion of the reported performance in literature may be attributed to simple memorization. This is further evidenced by the "naive Retention Heuristic." By simply looking up the most frequent relation for a given entity pair in the training data, this heuristic achieves a 90.9 F1 on CoNLL04's exact matches, nearly matching the performance of complex neural architectures like TABTO (92.6 F1).
Analysis: Joint vs. Pipeline Models
The study also reveals that architectural choices significantly impact a model's reliance on retention. Joint models, which optimize entity and relation extraction simultaneously, tend to be more prone to memorization and "forgetful" of context.
In the swapping experiment, the pipeline model PURE demonstrated superior robustness. While joint models like TABTO often predicted the original training triple (the "reverse RE" error) when arguments were swapped, PURE maintained a significantly higher F1 score on the swapped context. For example, in the "Kill" relation on CoNLL04, TABTO's performance plummeted from 89.6 F1 to 27.6 F1, while its reverse RE score was nearly 60%. PURE is likely more robust because it uses predicted entity types (e.g., "Person") as input for the relation classifier rather than raw lexical tokens, forcing the model to attend to linguistic triggers rather than memorized entity pairs.
Conclusion
The findings suggest that current RE benchmarks are heavily influenced by a model's ability to memorize training data. While joint models often achieve high holistic scores, they are more susceptible to retention bias. Pipeline models, by decoupling entity and relation steps, demonstrate better generalization to novel contexts. Future work should focus on creating benchmarks that specifically probe for context-based extraction to ensure models are truly learning to "read" rather than just "remember."
Summary updated