Supervising Unsupervised Open Information Extraction Models

Arpita Roy, Youngja Park, Taesung Lee, Shimei PanEMNLP 2019 · DOI

TL;DR

This paper introduces SenseOIE, a novel framework that combines multiple unsupervised Open Information Extraction (OpenIE) systems with a small amount of labeled data to improve extraction performance. By ensembling diverse unsupervised systems and leveraging rich linguistic features, SenseOIE significantly outperforms existing supervised and unsupervised OpenIE models, demonstrating a powerful approach to overcoming data scarcity in relation extraction.

What Problem Does This Paper Address?

Traditional supervised Information Extraction (IE) methods require extensive labeled data, which is expensive and time-consuming to create, especially for a large number of relations or new domains. Unsupervised OpenIE systems, on the other hand, can operate in a domain-independent manner and discover new relations without prior knowledge. However, existing unsupervised OpenIE systems often produce heterogeneous and inconsistent results due to their differing extraction rules and patterns, making it difficult to select the best system for a given task. This paper addresses the challenge of improving the performance and reliability of OpenIE by effectively combining the strengths of multiple unsupervised systems and leveraging a small amount of labeled data to guide the extraction process.

Methods

The core of the proposed approach, SenseOIE, is an ensemble of unsupervised OpenIE systems. Instead of relying on a single unsupervised system, SenseOIE takes the outputs from multiple such systems (Stanford OpenIE, OpenIE5, and UKG in this case) as input features. These features, representing the EOR (Entity 1, Relation, Entity 2, or Other) tags assigned by each system to every word, are combined with other rich linguistic features. These include:

  • Word embeddings: Capturing semantic meaning of words.
  • Part-of-speech (POS) embeddings: Indicating the grammatical role of words.
  • Syntactic role embeddings: Representing the grammatical function of words.
  • Dependency parse tree information: Specifically, the parent and immediate children of a word in the dependency tree.

This combined feature set is then fed into a bidirectional Long Short-Term Memory (Bi-LSTM) network. The Bi-LSTM is a standard deep learning architecture for sequence tagging tasks, chosen for its ability to capture context from both past and future words in a sentence. The model is trained to classify each word into one of the EOR tags, effectively learning to identify the relation phrases and their arguments. The use of a small amount of labeled data allows the model to be supervised and optimized, learning to combine the knowledge from the unsupervised systems and produce higher-quality tuples.

Datasets

The paper uses four benchmark datasets for training and evaluation:

  • AW-OIE: A large dataset for supervised OpenIE, created by extending the OIE2016 corpus.
  • AW-OIE-C: A cleaned version of AW-OIE, filtered to remove noisy and missing relations. This dataset is used for both training and testing.
  • WEB: A dataset designed to represent the challenges of dealing with noisy web text, including incomplete and grammatically unsound sentences.
  • NYT: Contains formal, well-written news stories from the New York Times Corpus.
  • PENN: Created from the Penn Tree Bank.

AW-OIE-C is used for training SenseOIE, while AW-OIE-C, WEB, NYT, and PENN are used for testing.

Key Insights and Ideas

  • Ensembling Unsupervised Systems is Powerful: The paper strongly suggests that combining the outputs of multiple diverse unsupervised OpenIE systems is more effective than relying on any single system. This ensemble approach helps to mitigate the weaknesses of individual systems and achieve broader coverage of extracted relations.
  • Leveraging Linguistic Features Enhances Performance: Beyond the raw outputs of unsupervised systems, incorporating rich linguistic features like word embeddings, POS tags, and dependency information significantly boosts the performance of the supervised model.
  • Small Labeled Datasets Can Be Highly Effective: The success of SenseOIE demonstrates that even a small amount of labeled data can be sufficient to supervise and optimize a complex model that leverages the knowledge from unsupervised sources, effectively addressing the data scarcity problem.
  • SenseOIE as a "Digital Annotator": The paper explores using SenseOIE to automatically generate annotated data for training new supervised models. The results show that a model trained on SenseOIE-generated labels performs comparably to one trained on human-labeled data, highlighting its potential for bootstrapping supervised IE in new domains.

Key Results

SenseOIE significantly outperforms all baseline systems, including both supervised (RnnOIE) and unsupervised (OpenIE5, Stanford OpenIE, UKG, Majority Votes) methods, by a large margin.

  • On the AW-OIE-C dataset, SenseOIE achieves a tuple-level F1-score of 0.79.
  • Compared to unsupervised OpenIE methods, SenseOIE shows performance gains ranging from 66% to 315%.
  • Specifically, SenseOIE outperforms OpenIE5 by 36% to 56% and UKG by 92% to 186%.
  • A feature ablation study shows that using all features (embeddings + OpenIE results) yields the best performance, with an average F1-score of 0.79 on AW-OIE-C.
  • The model trained on SenseOIE-generated labels achieves similar F1-scores to models trained on human-labeled data across the tested datasets, indicating its effectiveness as an annotator.

Limitations and Future Work

  • Dependency Tree Feature Effectiveness: The paper notes that a model without dependency parse tree features surprisingly outperforms SenseOIE in 2 out of 4 datasets. This suggests that the simple concatenation of dependency information might not be the optimal way to integrate it, and future work could explore more sophisticated methods for utilizing this feature.
  • Cold-Start Problem: While SenseOIE effectively addresses the cold-start problem by leveraging unsupervised systems, the paper suggests further research into more advanced techniques for automatically generating annotated data for new domains.
  • Domain Adaptation: The current work focuses on general OpenIE. Future directions could involve adapting SenseOIE to specific domains where specialized knowledge or relations are of interest.

Summary updated

BibTeX
@inproceedings{roy2019supervising,
  author = {Arpita Roy and Youngja Park and Taesung Lee and Shimei Pan},
  title = {{Supervising Unsupervised Open Information Extraction Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  doi = {10.18653/v1/d19-1067},
}
Figure