Let's Stop Incorrect Comparisons in End-to-end Relation Extraction!

Bruno Taillé, Vincent Guigue, Geoffrey Scoutheeten, patrick gallinariOpenReview 2020 · OpenReview

TL;DR

This paper tackles the pervasive issue of inconsistent and misleading performance comparisons in end-to-end Relation Extraction (RE) research. By identifying common pitfalls in evaluation methodologies and conducting a focused empirical study, the authors demonstrate how these inconsistencies can inflate reported results by up to 5% and call for standardized evaluation practices to foster more reliable progress in the field.

What Problem Does This Paper Address?

Relation Extraction (RE) is a crucial task in Information Extraction, aiming to identify relationships between entities in text. Traditionally, RE was part of a pipeline with Named Entity Recognition (NER), but end-to-end models that jointly handle both tasks have gained traction, especially with the advent of powerful language models like BERT. However, the field is plagued by a lack of standardized evaluation practices. Researchers often use different criteria for determining a "correct" prediction (e.g., strict boundary matching vs. looser criteria), employ varying dataset splits, or fail to report crucial dataset statistics. This makes it incredibly difficult to compare results across papers, leading to potentially inflated performance claims and hindering genuine scientific progress. The authors aim to bring clarity and rigor to this area by identifying these issues and proposing solutions.

Methods

The paper's contributions are multifaceted:

  1. Identification of Invalid Comparison Patterns: The authors meticulously reviewed existing literature to pinpoint recurring mistakes in how end-to-end RE papers compare their results. This includes issues like comparing "Strict" evaluation results against "Boundaries" evaluation results, or using different dataset splits without clear documentation.
  2. Empirical Study on Evaluation Impact: To quantify the effect of these inconsistencies, the researchers conducted a small empirical study. They specifically investigated the impact of using the "Boundaries" evaluation setting versus the "Strict" setting on the ACE05 dataset, estimating the overestimation in performance.
  3. Ablation Studies on Recent Developments: Leveraging their experimental setup, the authors performed ablation studies on two significant recent advancements: the use of BERT for language model pretraining and the adoption of span-level NER. This helps to disentangle the contributions of these components to overall performance gains.
  4. Call for Standardization: The paper advocates for more rigorous reporting of evaluation settings, dataset statistics, and ultimately, a unified evaluation framework for end-to-end RE.

Datasets

The paper primarily uses the following datasets for evaluation and experimentation:

  • CoNLL04: A widely used dataset for RE, annotated for four entity types and five relation types.
  • ACE05: Another standard dataset, defining seven coarse entity types and six relation types.
  • ADE: A biomedical dataset focusing on Adverse Drug Events.
  • SciERC: A dataset for scientific abstracts, featuring scientific entities and relations.

The authors emphasize the importance of reporting detailed dataset statistics (number of sentences, entity mentions, relation mentions, train/dev/test splits) to ensure reproducibility and prevent confusion, noting that such statistics are often missing or inconsistently reported in prior work.

Key Insights and Ideas

  • The "Boundaries" vs. "Strict" trap: Comparing results obtained with a "Boundaries" evaluation (where entity types don't need to be correct, only boundaries) against "Strict" evaluation (requiring both correct boundaries and types) can artificially inflate performance by 2.5-3 F1 points on datasets like ACE05. This is less pronounced on CoNLL04 due to its simpler relation structure.
  • BERT is the primary driver of gains: Ablation studies suggest that the significant performance improvements seen in recent end-to-end RE models are largely attributable to the use of pre-trained language models like BERT, rather than solely to architectural changes or new decoders.
  • Span-level NER's limited benefit on non-overlapping data: While span-level NER is designed to handle overlapping entities, its advantage over traditional sequence tagging NER for end-to-end RE on datasets without overlapping mentions (like CoNLL04 and ACE05) is not evident from their experiments.
  • Lack of reproducibility fuels errors: The difficulty in reproducing prior work, often due to missing source code and detailed experimental setups, makes it hard to catch and correct evaluation errors, perpetuating flawed comparisons.

Key Results

  • The study quantifies that using the "Boundaries" evaluation setting instead of "Strict" can lead to an overestimation of RE performance by 2.5 to 3 F1 points on ACE05.
  • Their baseline model, using BERT and span-level NER, achieved competitive results, but ablation studies showed that replacing BERT with a BiLSTM encoder led to a significant performance drop, highlighting BERT's crucial role.
  • The experiments also indicated that span-level NER did not provide a benefit over BILOU sequence tagging for NER in their end-to-end RE setup on non-overlapping datasets.

Limitations and Future Work

The authors acknowledge that their empirical study is small and focuses on a limited set of models and datasets. They also note that reproducing prior work is challenging due to the complexity of end-to-end RE models and the frequent absence of documented source code.

Future work suggested by the paper includes:

  • Unified Evaluation Setting: A strong call to action for the community to adopt a standardized evaluation framework, advocating for the "Strict" criterion as the primary metric, supplemented by "Boundaries" for more nuanced analysis.
  • More Rigorous Reporting: Encouraging authors to provide detailed dataset statistics and clear descriptions of their evaluation settings to enhance reproducibility.
  • Further Studies: Suggesting that future research should explore the impact of lexical overlap and shallow heuristics in neural RE models, and continue to investigate the behavior of models across different datasets and settings.

Summary updated

Abstract

Despite efforts to distinguish three different evaluation setups (Bekoulis et al., 2018), numerous end-to-end Relation Extraction (RE) articles present unreliable performance comparison to previous work. In this paper, we first identify several patterns of invalid comparisons in published papers and describe them to avoid their propagation. We then propose a small empirical study to quantify the impact of the most common mistake and evaluate it leads to overestimating the final RE performance by around 5% on ACE05. We also seize this opportunity to study the unexplored ablations of two recent developments: the use of language model pretraining (specifically BERT) and span-level NER. This meta-analysis emphasizes the need for rigor in the report of both the evaluation setting and the datasets statistics and we call for unifying the evaluation setting in end-to-end RE.
BibTeX
@misc{taille2020let,
  author = {Bruno Taillé and Vincent Guigue and Geoffrey Scoutheeten and patrick gallinari},
  title = {{Let's Stop Incorrect Comparisons in End-to-end Relation Extraction!}},
  year = {2020},
  eprint = {2009.10684},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2009.10684},
  howpublished = {OpenReview},
}
Figure