Joint Bootstrapping Machines for High Confidence Relation Extraction

Pankaj Gupta, Benjamin Roth, Hinrich Schütze

Abstract

Semi-supervised bootstrapping techniques for relationship extraction from text iteratively expand a set of initial seed instances. Due to the lack of labeled data, a key challenge in bootstrapping is semantic drift: if a false positive instance is added during an iteration, then all following iterations are contaminated. We introduce BREX, a new bootstrapping method that protects against such contamination by highly effective confidence assessment. This is achieved by using entity and template seeds jointly (as opposed to just one as in previous work), by expanding entities and templates in parallel and in a mutually constraining fashion in each iteration and by introducing higher-quality similarity measures for templates. Experimental results show that BREX achieves an F1 that is 0.13 (0.87 vs. 0.74) better than the state of the art for four relationships.

Introduction

Relation extraction (RE) is the task of identifying structured relationships between entities in plain text—for example, recognizing that the phrase "Google acquired YouTube" implies an acquired relationship. While supervised learning is the gold standard for this task, it requires massive amounts of hand-labeled data. To avoid this bottleneck, researchers use bootstrapping: starting with a few "seed" examples (like <Google, YouTube>), the system identifies their occurrences in a corpus, learns the surrounding patterns, and uses those patterns to discover new entity pairs.

The fatal flaw of bootstrapping is semantic drift. If a system accidentally learns a vague or incorrect pattern (like "X and Y"), it might extract a false positive (like <Apple, Microsoft>). In subsequent iterations, the system uses that error to find even more incorrect patterns, eventually "drifting" away from the target relationship until the extracted data is heavily contaminated.

Methods

The authors introduce BREX (Bootstrapping RElation eXtraction), which tackles semantic drift through three main innovations: joint learning, a multi-stage expansion process, and advanced similarity measures.

Joint Learning (BREJ)

Unlike previous systems that bootstrap using either entity pairs (BREE) or text templates (BRET) in isolation, BREX uses both jointly. By requiring instances to satisfy constraints from both sides, the system "scales up" the number of high-confidence positive matches. This joint approach acts as a mutual filter, effectively drowning out the noisy, low-confidence extractions that typically cause semantic drift.

Comparison of BREE, BRET, and the joint BREJ approach in generating extractors and output instances.

The Three-Hop Expansion

To ensure high recall without sacrificing precision, the algorithm looks for neighbors in three distinct stages:

  1. Hop-1: Identifies instances directly similar to the initial seeds.
  2. Hop-2: Clusters these instances to create generalized "extractors."
  3. Hop-3: Finds all other instances in the corpus that match these new extractors.
Overview of the BREX three-hop expansion process.

Cross-Context Attentive Similarity

Traditional systems often compare the text before, between, and after entities separately. BREX introduces a flexible similarity measure ($sim_{cc}$) that can match words across these boundaries. This allows the system to recognize that "X's acquisition of Y" is semantically similar to "X acquired Y," even though the keywords appear in different parts of the sentence structure.

Experimental Results

The researchers evaluated BREX using a corpus of 5.5 million news articles from the English Gigaword corpus. They used FreebaseEasy as a ground-truth knowledge base to calculate precision and recall for four target relations: acquired, founder-of, headquartered, and affiliation.

  • F1 Score Boost: The joint model (BREJ) achieved an average F1 score of 0.87, compared to the previous state-of-the-art (BREDS) at 0.74. This represents a 13% absolute gain.
  • Confidence Scoring: The paper defines a sophisticated confidence score that penalizes extractors that pull in known "negative" seeds (e.g., if a pattern for founders pulls in competitors, that pattern's confidence is slashed).
  • Parameter Efficiency: The new similarity measures allowed the researchers to eliminate four hyper-parameters used in previous models, making the system easier to tune and more generalizable.

Conclusion

BREX demonstrates that joint learning from entity and template signals provides a robust defense against semantic drift. While the system still depends on a small set of high-quality seeds and accurate named entity tagging, it significantly advances the state of the art in semi-supervised relation extraction. Future work may extend this method to more complex, non-binary relationships or domain-specific extractions where labeled data is particularly scarce.

Summary updated

Abstract

Semi-supervised bootstrapping techniques for relationship extraction from text iteratively expand a set of initial seed instances. Due to the lack of labeled data, a key challenge in bootstrapping is semantic drift: if a false positive instance is added during an iteration, then all following iterations are contaminated. We introduce BREX, a new bootstrapping method that protects against such contamination by highly effective confidence assessment. This is achieved by using entity and template seeds jointly (as opposed to just one as in previous work), by expanding entities and templates in parallel and in a mutually constraining fashion in each iteration and by introducing higherquality similarity measures for templates. Experimental results show that BREX achieves an F1 that is 0.13 (0.87 vs. 0.74) better than the state of the art for four relationships.
BibTeX
@inproceedings{gupta2018joint,
  author = {Pankaj Gupta and Benjamin Roth and Hinrich Schütze},
  title = {{Joint Bootstrapping Machines for High Confidence Relation Extraction}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2018},
  url = {https://openreview.net/forum?id=ByZajMZOZr},
}
Figure