Abstract
Open information extraction (Open IE) is a challenging task especially due to its brittle data basis. Most Open IE systems have to be trained on automatically built corpora and evaluated on inaccurate test sets. In this work, we first alleviate this difficulty from both sides of training and test sets. For the former, we propose an improved model design to more sufficiently exploit training datasets. For the latter, we present our accurately re-annotated benchmark test set (Re-OIE2016) according to a series of linguistic observations and analysis. Then, we introduce a span model instead of previous adopted sequence labeling formulization for n-ary Open IE. Our newly introduced model achieves new state-of-the-art performance on both benchmark evaluation datasets.
Introduction
Open Information Extraction (Open IE) is the task of automatically pulling structured "tuples"—typically a predicate and its arguments—from unstructured text without relying on a pre-defined schema. Despite advancements in neural models, the field faces two major roadblocks: inaccurate, noisy training data and the limitations of token-level sequence labeling (such as BIO tagging), which struggles to capture the features of entire phrases (spans) effectively. This paper introduces SpanOIE, the first span-based model for n-ary Open IE, alongside a manually cleaned benchmark, Re-OIE2016.
Methodology: The SpanOIE Architecture
SpanOIE shifts the paradigm from token-level classification to span selection. The model operates as a two-stage pipeline: a Predicate Module identifies potential predicate spans, and an Argument Module selects the most likely spans for various argument roles (A0, A1, A2, etc.) for a given predicate.
By scoring all possible text spans $(i, j)$, the model can utilize span-level features, such as the difference between start and end word vectors. To manage the computational complexity of evaluating all spans, the authors implement syntactic pruning using dependency trees and length constraints. A key innovation is the use of a weighted loss function; rather than filtering out low-confidence extractions from the training set, the model weights the loss by the source extraction's confidence score, allowing it to learn from a broader range of linguistic patterns while minimizing the impact of errors.
Dataset Refinement: Re-OIE2016
To address the "brittle" evaluation standards in Open IE, the authors manually re-annotated the 600-sentence OIE2016 test set. This new benchmark, Re-OIE2016, fixes common errors where verbal adjectives were mislabeled as predicates and adds missing appositive relations, providing a more reliable metric for system performance.
Experimental Results
SpanOIE achieved new state-of-the-art (SOTA) performance on both the original and the re-annotated benchmarks. On the original OIE2016, it achieved an Area Under the Curve (AUC) of 0.489. On the cleaner Re-OIE2016, the model's performance significantly improved, reaching an AUC of 0.659 and an F1 score of 78.50.
Qualitative analysis indicates that SpanOIE is superior at identifying argument boundaries and correctly extracting complex n-ary relations with multiple modifiers, such as "at reduced prices," which traditional binary models often fail to capture.
Conclusion
By moving from tokens to spans, SpanOIE effectively addresses the boundary detection issues inherent in sequence labeling models. Combined with a more accurate benchmark and a confidence-weighted training approach, this work provides a more robust foundation for future research in Open Information Extraction.
Summary updated