Span Model for Open Information Extraction on Accurate Corpus

Junlang Zhan, Hai Zhao

Abstract

Open Information Extraction (Open IE) aims to extract structured n-ary tuples from unstructured text without a pre-defined schema. Traditional neural approaches often rely on sequence labeling (e.g., BIO tagging), which treats tokens individually and struggles to capture phrase-level features or complex n-ary relations. We present SpanOIE, the first span-based model for Open IE. By reframing extraction as a span selection task and introducing a manually re-annotated benchmark, Re-OIE2016, we demonstrate significant improvements in extraction accuracy. SpanOIE achieves state-of-the-art performance, highlighting the importance of modeling phrases directly and utilizing high-quality evaluation data.

Introduction

Open Information Extraction (Open IE) is a critical task for automatically pulling structured tuples—typically (Subject; Predicate; Object)—from unstructured text. These extractions are vital for downstream applications such as knowledge base construction and question answering. However, current Open IE systems face two primary hurdles: the prevalence of noisy "silver" training data and the architectural limitations of token-level sequence labeling models.

Sequence labeling models treat every word as an individual unit, making it difficult to capture the semantic and syntactic features of an entire phrase (a "span"). Furthermore, these models often struggle with n-ary relations, where a single predicate may have more than two arguments. To address these issues, we propose SpanOIE, a model that treats extraction as a span selection task, and provide a more accurate benchmark for the community.

Methodology

SpanOIE utilizes a two-stage pipeline to extract facts from natural language sentences.

Two-Stage Pipeline

The extraction process begins with a predicate module that identifies potential predicates in a sentence. For each identified predicate, an argument module then scans the sentence to identify corresponding argument spans (e.g., A0, A1, A2). This approach naturally supports n-ary extraction by allowing the model to select multiple non-overlapping spans for a single predicate.

Span Selection and Feature Engineering

Instead of labeling tokens, SpanOIE evaluates all possible spans $(i, j)$ within a sentence and scores them for specific roles. To maintain computational efficiency, the model applies constraints: spans cannot exceed a fixed length, cannot overlap with the predicate, and must satisfy certain syntactic parent-child relationships.

The SpanOIE architecture constructs span features by concatenating BiLSTM hidden states of the start and end points, along with their sum and difference.

As illustrated in the architecture above, the model employs a BiLSTM to create contextualized word representations. Span features are then constructed by concatenating the hidden states of the span's start and end points, along with their element-wise sum and difference. This representation allows the model to exploit features like the "syntactic head" of a phrase, which is often lost in token-level models.

Robust Training

To handle noisy training data, the authors modified the loss function to weight training samples by their confidence scores from the source extractor (OpenIE4). This "smart training" strategy ensures the model learns primarily from high-quality examples while still benefiting from the breadth of the large-scale bootstrapped corpus.

Datasets and Evaluation

The model was trained on a corpus of 1.1 million sentences bootstrapped from Wikipedia. Evaluation was conducted on the standard OIE2016 benchmark. However, recognizing systematic errors in OIE2016—such as the mislabeling of verbal adjectives and the omission of appositive relations—the authors introduced Re-OIE2016. This manually re-annotated test set provides a more rigorous and accurate measure of Open IE performance.

Results

SpanOIE significantly outperformed existing systems across both benchmarks. On the original OIE2016 set, SpanOIE achieved an AUC of 0.489, surpassing the previous best neural model, RnnOIE, and the widely used OpenIE4 (0.408).

Precision-Recall curves on the Re-OIE2016 benchmark demonstrate that SpanOIE significantly outperforms traditional sequence labeling and rule-based systems.

On the more accurate Re-OIE2016 benchmark, the performance gap widened further. SpanOIE reached an AUC of 0.659 and an F1 score of 78.50, compared to OpenIE4’s 0.509 AUC. Qualitative analysis shows that the model is superior at identifying "hidden" predicates and correctly defining argument boundaries, particularly in complex sentences involving pronouns or participial phrases.

Conclusion

SpanOIE represents a shift in Open IE from token-level labeling to span-level selection. By modeling phrases directly, the system captures more nuanced information and achieves state-of-the-art results. While challenges remain in extracting attributional context and perfectly integrating syntactic features, SpanOIE and the Re-OIE2016 benchmark provide a new foundation for more accurate and robust Open Information Extraction.

Summary updated

Abstract

In this paper, we propose a span based model combined with syntactic information for n-ary open information extraction. The advantage of span model is that it can leverage span level features, which is difficult in token based BIO tagging methods. We also improve the previous bootstrap method to construct training corpus. Experiments show that our model outperforms previous open information extraction systems. Our code and data are publicly available at this https URL
BibTeX
@inproceedings{zhan2020span,
  author = {Junlang Zhan and Hai Zhao},
  title = {{Span Model for Open Information Extraction on Accurate Corpus}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i05.6497},
}
Figure