Abstract
Open Information Extraction (OIE) aims to distill unstructured text into structured (subject; relation; object) tuples. However, existing benchmarks often suffer from lenient token-level matching or rigid, error-prone annotation sets. This paper introduces BenchIE$^{FL}$, a manually re-annotated benchmark that addresses these quality issues through stricter guidelines and a flexible matching function. BenchIE$^{FL}$ demonstrates a significantly higher correlation with downstream task performance compared to previous benchmarks, providing a more reliable metric for OIE system evaluation.
Introduction
Evaluating OIE systems is notoriously difficult. Early benchmarks relied on "token-level" matching, which rewarded systems for simple word overlap with ground truth, even if the resulting tuples were logically incoherent. While the recent BenchIE benchmark introduced "exact matching" against fact clusters to improve rigor, it was found to be overly rigid and riddled with data quality issues. Analysis revealed that over 50% of BenchIE's sentences were missing facts, while others contained false or irrelevant information. Consequently, a system’s benchmark score often failed to predict its utility in real-world applications like Question Answering (QA).
Methodology
BenchIE$^{FL}$ overhauls the OIE evaluation framework through two primary pillars: refined annotation principles and a flexible matching heuristic.
New Annotation Guidelines
The authors re-annotated 300 sentences based on five core principles designed to ensure high-quality, usable extractions:
- Informativity: Tuples must contain meaningful, specific information.
- Minimality: Facts are decomposed into their smallest possible components.
- Exhaustivity: Every piece of information in the source sentence must be captured.
- Relation Completeness: Relations must be self-contained and accurate.
- Light Inference: Facts clearly implied by the text are included, moving beyond strict verbatim extraction.
Flexible Matching Function
To move beyond the "all-or-nothing" approach of exact matching, BenchIE$^{FL}$ employs a matching function that uses heuristics to credit valid extractions. This function accounts for alternative formulations, varying levels of detail, and trivial differences in punctuation or capitalization, achieving a human-like matching F1 score of 0.97.
Evaluation and Key Results
The BenchIE$^{FL}$ dataset contains 1,798 fact clusters derived from 300 sentences, increasing the average fact density from 4.5 to 6.0 clusters per sentence compared to the original BenchIE. The authors evaluated seven OIE systems, ranging from rule-based veterans (ReVerb, ClausIE, MinIE) to modern neural models (IMojIE, OpenIE6, M2OIE, CompactIE).
Key Insights
- The Neural "Copy-Paste" Problem: Modern neural OIE systems often "cheat" on older metrics by copying large chunks of input text into a single tuple. While this yields high overlap scores, it produces "noisy" extractions that are difficult for downstream logic to process.
- The Resilience of Rule-Based Systems: Older rule-based systems like MinIE often outperform state-of-the-art neural models on BenchIE$^{FL}$ because they are inherently better at producing minimal, precise facts.
- Downstream Predictive Power: The most significant finding is the correlation with downstream tasks. While previous benchmarks like CaRB showed a negative correlation with task performance, BenchIE$^{FL}$ achieved a 0.940 Pearson correlation with Assertion-Based QA (ABQA) performance. This indicates that a high score on BenchIE$^{FL}$ finally serves as a reliable proxy for real-world system utility.
Conclusion
BenchIE$^{FL}$ provides a much-needed rigorous standard for OIE evaluation. By prioritizing fact-based accuracy and exhaustive annotation, it bridges the gap between benchmark scores and downstream performance. While currently limited to 300 English sentences, future work aims to scale these high-quality annotations to larger datasets and multiple languages, potentially using BenchIE$^{FL}$ as a gold standard to train more robust automated OIE extractors.
Summary updated