Span Based Open Information Extraction

Junlang Zhan, Hai Zhao

Abstract

In this paper, we propose a span-based model combined with syntactic information for n-ary open information extraction. The advantage of the span model is that it can leverage span-level features, which is difficult in token-based BIO tagging methods. We also improve the previous bootstrap method to construct training corpus. Experiments show that our model outperforms previous open information extraction systems. Our code and data are publicly available.

Introduction

Open Information Extraction (Open IE) is the task of distilling structured "facts"—usually in the form of (Subject; Predicate; Argument)—from unstructured text. While it is a cornerstone of NLP, the field has long been plagued by "brittle" data: training sets are often automatically generated and noisy, and evaluation benchmarks frequently contain errors.

Most modern supervised Open IE systems rely on BIO tagging (Beginning, Inside, Outside). In this setup, a model decides for every single word whether it belongs to an argument or a predicate. However, this approach struggles to capture "span-level" features—the holistic properties of a multi-word phrase. Researchers from Shanghai Jiao Tong University introduce SpanOIE, a model that shifts the paradigm from token-level sequence labeling to span-level selection, paired with a more robust training strategy and a cleaned-up benchmark.

The SpanOIE Architecture

SpanOIE treats Open Information Extraction as a span-selection task rather than traditional BIO tagging, allowing it to capture richer phrase-level features. The model follows a two-stage pipeline:

  1. Predicate Module: Identifies potential predicate spans in a sentence.
  2. Argument Module: Given a sentence and a specific predicate, it selects the most likely spans for arguments (A0, A1, A2, etc.).
Figure 1: The SpanOIE architecture uses BiLSTM layers to generate span representations based on start and end token vectors, including their sum and difference.

Instead of labeling tokens, the model scores all possible spans $(i, j)$ in a sentence. By representing a phrase as a combination of its start/end word vectors and their differences, the model captures "boundary" information much more effectively than a token-by-token BIO tagger. To keep the computational cost manageable, the authors apply constraints: argument spans must be less than 10 words, and predicate spans less than 5 words.

Weighted Loss and Syntactic Integration

Supervised models need massive amounts of data. Since human annotation is expensive, researchers often use "bootstrap" methods where an older, rule-based system (like OpenIE4) labels a large corpus to train a newer neural model. Previous work simply discarded extractions with low confidence scores. The authors argue this is wasteful: some low-confidence extractions are actually correct, while some high-confidence ones are wrong.

SpanOIE uses a Weighted Loss Function. Instead of a hard threshold, they use a "soft" approach, multiplying the cross-entropy loss of each training sample by the confidence score assigned to it by the source extractor. This allows the model to learn from more data while naturally prioritizing reliable samples. This approach particularly helped the model learn to extract pronoun arguments (e.g., "he," "it") which source extractors often flagged with low confidence.

Benchmarking and Results

The standard evaluation set, OIE2016, is riddled with issues like overspecification and missing appositive relations. To address this, the authors introduced Re-OIE2016, a manually re-annotated version of the 600-sentence test set that fixes errors in predicate identification and properly handles appositives and conjunctions.

Figure 3: Precision-Recall curves on the Re-OIE2016 benchmark show SpanOIE (top blue line) consistently outperforming previous systems.

SpanOIE consistently outperformed previous systems like ClausIE, OpenIE4, and RnnOIE across both benchmarks. The jump in performance on Re-OIE2016 is particularly striking, suggesting that when evaluated on accurate data, the advantages of the span-based model become even more apparent. On the Re-OIE2016 benchmark, SpanOIE achieved an AUC of 0.659, compared to 0.579 for a standard BIO baseline and 0.509 for OpenIE4.

Conclusion

By moving from token-level tagging to span-level selection and implementing a confidence-weighted loss function, SpanOIE sets a new state-of-the-art for Open Information Extraction. While challenges remain in handling complex context boundaries and appositive relations, the shift toward span-level modeling provides a more robust foundation for distilling structured knowledge from natural language.

Summary updated

Abstract

In this paper, we propose a span based model combined with syntactic information for n-ary open information extraction. The advantage of span model is that it can leverage span level features, which is difficult in token based BIO tagging methods. We also improve the previous bootstrap method to construct training corpus. Experiments show that our model outperforms previous open information extraction systems. Our code and data are publicly available at this https URL
BibTeX
@inproceedings{zhan2020span,
  author = {Junlang Zhan and Hai Zhao},
  title = {{Span Based Open Information Extraction}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i05.6497},
}
Figure