Abstract
Open Information Extraction (Open IE) is a critical task for converting unstructured text into structured tuples. However, existing systems often struggle with efficiency, capturing complex semantic relationships, and generalizing across languages. We present Multi$^2$OIE, a high-performance, multilingual Open IE system that utilizes a combination of BERT and multi-head attention (MHA) to replace traditional recurrent architectures. Multi$^2$OIE achieves state-of-the-art results for sequence-labeling models, demonstrating a 14x speedup over generation-based alternatives and robust zero-shot extraction capabilities in non-English languages.
Introduction
Open Information Extraction (Open IE) aims to extract structured (Subject; Relation; Object) tuples from unstructured text without a predefined schema. For instance, the sentence "The candidate was elected President" yields the tuple (was elected; The candidate; President). Despite its utility, state-of-the-art Open IE models face three primary challenges:
- The Efficiency Gap: Generative models, while accurate, are slow due to their autoregressive nature.
- The Context Gap: Faster sequence-labeling models often fail to capture deep semantic links between predicates and arguments.
- The Language Gap: Most systems are English-centric, and building multilingual models is hindered by a lack of annotated data in other languages.
Multi$^2$OIE addresses these gaps by adopting a sequence-labeling approach enhanced by Multi-Head Attention and multilingual BERT (mBERT).
Methodology
Multi$^2$OIE operates as a two-stage sequence-labeling system: Predicate Extraction followed by Argument Extraction.
Predicate Extraction
The model first identifies the "relation" or predicate within a sentence using BERT. For a sentence like "The man was born in 1960," the model identifies "was born" as the predicate.
Argument Extraction via Multi-Head Attention (MHA)
The core innovation of Multi$^2$OIE is the use of MHA blocks for argument extraction, moving beyond simple concatenation. In this setup, the model treats the sentence and the predicate as two different modalities to be fused.
As shown in the architecture, the Query represents the hidden states of the entire sentence, while the Key and Value represent the hidden states specifically corresponding to the identified predicate's position. This allows the model to "attend" to the predicate's context while assigning BIO tags (e.g., ARG0, ARG1) to every other word in the sentence. By leveraging multilingual BERT (mBERT), the system projects these relationships into a shared language space, facilitating cross-lingual transfer.
Experimental Setup
The model was trained on the OpenIE4 dataset, comprising approximately 1.1 million sentences. For English evaluation, the authors utilized the Re-OIE2016 and CaRB benchmarks. To evaluate multilingual performance, the authors introduced new benchmarks for Spanish and Portuguese by translating and manually re-annotating the Re-OIE2016 dataset to ensure high-quality ground truth.
Results and Discussion
Performance and Efficiency
Multi$^2$OIE significantly outperformed previous sequence labelers, achieving a 6.9%p F1 improvement on Re-OIE2016 and a 2.9%p F1 improvement on CaRB compared to SpanOIE. Crucially, it maintains high efficiency; Multi$^2$OIE was 14 times faster than the generative IMoJIE model, processing 641 sentences in 15.5 seconds, whereas IMoJIE required over 3 minutes.
Zero-Shot Multilingual Extraction
A key finding is the model's "zero-shot" capability. Despite being trained solely on English data, Multi$^2$OIE outperformed existing rule-based multilingual systems like ArgOE and PredPatt on Spanish and Portuguese benchmarks. This success is attributed to mBERT's ability to generate language-agnostic embeddings, allowing the model to generalize learned extraction patterns across languages.
Conclusion
Multi$^2$OIE demonstrates that combining sequence labeling with multi-head attention and pre-trained multilingual transformers can bridge the gaps in efficiency, context, and language for Open IE. While the model currently struggles with nominal relations and complex conjunctions, it provides a scalable and robust framework for web-scale multilingual information extraction. Future work will explore its performance on non-alphabetic languages like Chinese and Japanese.
Summary updated