Let's Stop Incorrect Comparisons in End-to-end Relation Extraction!

Bruno Taillé, Vincent Guigue, Geoffrey Scoutheeten, patrick gallinariEMNLP 2020 · OpenReview

Abstract: Bruno Taillé, Vincent Guigue, Geoffrey Scoutheeten, Patrick Gallinari. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).

2020.

Introduction

End-to-end Relation Extraction (RE) is often considered the "holy grail" of Information Extraction, requiring a single model to simultaneously identify entities (NER) and the semantic relationships between them. However, evaluating these models is complex. This paper identifies a critical issue in the field: researchers are frequently "comparing apples to oranges" by using inconsistent evaluation metrics, leading to a propagation of erroneous results and inflated performance claims.

The Evaluation Pitfall

The authors highlight three distinct evaluation settings used in the community: Strict, Boundaries, and Relaxed.

  • Strict: Requires correct entity boundaries and correct entity types.
  • Boundaries: Requires only correct entity boundaries.
  • Relaxed: Uses more lenient matching criteria.

Many recent papers claim superiority by comparing their "Boundaries" scores (which are easier to achieve) against a predecessor's "Strict" scores. As shown in the literature meta-analysis, this discrepancy can inflate performance by up to 5% relative F1, enough to make a mediocre model appear to be a new state-of-the-art.

Comparison of RE results across different evaluation criteria (Strict, Boundaries, Relaxed) showing how inconsistent metrics lead to unfair comparisons.

Dissecting Performance Drivers

To determine what actually drives recent gains, the authors conducted a "double ablation" study using a modern baseline (SpERT). They isolated two main factors: the encoder (BiLSTM vs. BERT) and the NER decoder (traditional BILOU sequence tagging vs. modern Span-level classification).

Ablation study results comparing BERT vs. BiLSTM encoders and Span-based vs. Sequence-based decoders across CoNLL04 and ACE05 datasets.

Key Findings

  • BERT is the Real Hero: Almost all recent empirical gains are driven by Language Model (LM) pretraining. On the ACE05 dataset, moving from a BiLSTM to BERT improved the RE Strict score from 50.3 to 59.7 F1—a massive jump that dwarfs architectural tweaks.
  • Span-level NER is Overhyped: Surprisingly, the study found no evidence that Span-level NER outperforms traditional BILOU sequence tagging for end-to-end RE. In many cases, sequence tagging yielded slightly better results.
  • The Metric Gap: The same BERT-based model scored 59.6 on the Strict metric but 62.9 on the Boundaries metric on ACE05. This 3.3-point gap is precisely the kind of "incorrect comparison" the authors warn against.

Dataset Inconsistencies

Beyond evaluation metrics, the authors point out that different papers reporting on the "same" dataset often use different preprocessing scripts. This results in significant variations in sentence and entity counts (e.g., one version of ACE05 having 10k sentences while another has 14k), making cross-paper comparisons even more unreliable.

Conclusion and Recommendations

To restore scientific rigor to the field, the authors call for the community to:

  1. Always report the "Strict" criterion as the primary metric.
  2. Report detailed dataset statistics (number of sentences, entities, and relations) to ensure data consistency.
  3. Unify evaluation settings to enable meaningful cross-dataset analysis and stop the fragmentation of the field.

Summary updated

BibTeX
@inproceedings{taille2020let,
  author = {Bruno Taillé and Vincent Guigue and Geoffrey Scoutheeten and patrick gallinari},
  title = {{Let's Stop Incorrect Comparisons in End-to-end Relation Extraction!}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=bR42fEg4KZa},
}
Figure