Seed Selection for Distantly Supervised Web-Based Relation Extraction

Isabelle Augenstein

TL;DR

This paper tackles the challenge of noisy training data in distant supervision for relation extraction. They propose and evaluate methods to strategically filter unreliable training examples, demonstrating a significant 35% reduction in errors and improving precision by strategically selecting "seed" data.

What Problem Does This Paper Address?

Relation extraction aims to identify and classify relationships between entities in text (e.g., "The Beatles" are from "Liverpool"). A common approach is "distant supervision," which automatically labels training data by aligning text with existing knowledge bases like Freebase. However, this method suffers from two main problems:

  1. Ambiguity: A single entity mention might refer to different things (e.g., "Let It Be" can be an album or a song), leading to incorrect labels.
  2. Incompleteness: Knowledge bases are not exhaustive; true relations might exist in text but not be recorded, causing distant supervision to mislabel them as negative examples.

These issues lead to noisy training data, which degrades the performance of relation extraction models. This paper focuses on improving the quality of this training data by carefully selecting reliable "seed" examples.

Methods

The core contribution of this paper lies in its proposed seed selection strategies to filter unreliable training data. They introduce several measures:

  • Ambiguity Within An Entity (Unam): This method discards lexicalizations (textual mentions) of objects if they are ambiguous for a given subject entity. For example, if "Let It Be" can refer to both an album and a track by "The Beatles," it's considered ambiguous and potentially filtered. They quantify ambiguity by counting the number of different senses a lexicalization has for a given subject.
  • Ambiguity Across Classes (Stop & Stat):
    • Stop: Filters out common "stop words" that are used as object lexicalizations, as these are often highly ambiguous.
  • Stat: Estimates cross-class ambiguity by comparing the number of senses of an object lexicalization to those of other objects within the same relation. If an object's ambiguity is low compared to others, it's considered more reliable.
  • Incompleteness (Incomp): This method attempts to discard negative training examples that are likely true relation mentions but are missing from the knowledge base. It checks if a potential negative example's object lexicalization appears with other properties of the same class.

These filtering methods are applied before training a classifier. The system uses a Conditional Random Field (CRF) model for classification, trained on a corpus derived from Freebase and Web pages.

Datasets

The authors created a corpus for Web relation extraction by querying the Freebase API for entities from three specific classes (Book author, Musical Artist, Politician) and their associated properties. They retrieved Web pages using Google Search API based on these entities and properties. The corpus was split into training and test sets. A portion of the test corpus, specifically instances with "NONE" predictions, was manually annotated to evaluate the system's performance on potentially missing relations.

Key Insights and Ideas

  • Strategic Filtering is Key: The paper strongly suggests that improving the quality of training data through careful seed selection is more effective than simply using more data or more complex models.
  • Ambiguity is a Major Hurdle: The "Unam" method, which targets ambiguity within an entity, proved to be a significant contributor to improved precision.
  • Negative Data is Tricky: The "Incomp" method, designed to handle missing information in knowledge bases, surprisingly degraded performance, suggesting that simply assuming potential false negatives are true relations is not a robust strategy.
  • Confidence Matters: The confidence scores assigned by the relation extraction model are shown to be well-calibrated, meaning higher confidence generally correlates with higher precision.

Key Results

The paper reports significant improvements in precision:

  • The best-performing model ("unam_stop_stat25") achieved a total precision of 0.896, compared to a baseline model's precision of 0.825.
  • This represents a total error reduction of 35%.
  • The precision-recall curves (Figure 1) show that as confidence increases, precision rises for all models, validating the confidence measure's effectiveness in ranking results.
  • Precision@K curves (Figure 2) also demonstrate that filtering methods generally improve precision for the top-ranked results.

Limitations and Future Work

Limitations:

  • The "Incomp" method for handling incomplete knowledge bases was unsuccessful and actually harmed performance.
  • The authors acknowledge that while their models achieve good precision, distant supervision inherently limits recall because it relies on sentences explicitly mentioning both entities.
  • Direct comparison to other distant supervision approaches is difficult due to differences in evaluation data and relation sets.

Future Work:

  • Increase the number of extractions by exploring methods beyond explicit co-occurrence in sentences, such as extracting from lists and relational tables on Web pages.
  • Extend the approach to entity classification to aid relation extraction.
  • Investigate ways to further exploit background knowledge to improve precision.

Summary updated

BibTeX
@inproceedings{augenstein2014seed,
  author = {Isabelle Augenstein},
  title = {{Seed Selection for Distantly Supervised Web-Based Relation Extraction}},
  booktitle = {SWAIE @ COLING},
  year = {2014},
  doi = {10.3115/v1/w14-6203},
}
Figure