A High Accuracy Method for Semi-Supervised Information Extraction

Stephen Tratz, Antonio SanfilippoNAACL 2007

TL;DR

This paper presents a novel semi-supervised learning approach for Information Extraction (IE) that significantly boosts accuracy by integrating fully-supervised techniques. By cleverly using a maximum entropy classifier to rank candidate patterns, the method achieves state-of-the-art results without increasing the user's annotation burden.

What Problem Does This Paper Address?

Customizing Information Extraction (IE) systems to specific domains or user needs is a major hurdle. Traditional methods, like rule-based systems or fully supervised machine learning, require substantial manual effort from users to create extraction patterns and ontologies. While unsupervised or weakly supervised methods aim to reduce this burden, they often fall short in accuracy, making them unsuitable for production systems. This paper tackles the challenge of achieving high accuracy in IE customization while minimizing user effort.

Methods

The core of this paper's contribution lies in its enhanced semi-supervised learning algorithm. It builds upon the work of Stevenson and Greenwood (2005) but introduces a key innovation: using a supervised classifier to rank candidate patterns.

Here's a breakdown of their approach:

  • Candidate Pattern Generation: The system starts with an unannotated corpus and a small set of "seed" patterns (examples of desired extraction patterns). It then extracts features from sentences and identifies potential candidate patterns.
  • Active Learning for Training Data: Instead of manual annotation, the system uses an active learning strategy. It selects patterns with the lowest similarity to seed patterns as negative training examples and high-similarity patterns as positive examples. This is based on the hypothesis that low-similarity patterns are likely uninformative.
  • Maximum Entropy Classifier: A maximum entropy classifier is trained on these automatically generated positive and negative examples. This classifier learns to weigh different features, providing a more robust and flexible way to rank candidate patterns compared to relying solely on semantic similarity.
  • Iterative Refinement: The top-ranked patterns are added to the set of "accepted" patterns, and the process iterates. This iterative refinement allows the system to gradually discover and learn more accurate extraction patterns.
  • Rich Feature Set: The system utilizes a comprehensive set of features, including single tokens (with POS tags and dependencies), token combinations (bigrams, subject-object pairs), and semantic information (like named entity types). This richer feature set, combined with the classifier's ability to weigh them, is crucial for improved performance.

Datasets

The paper uses the MUC-6 (Message Understanding Conference 6) data set for evaluation. This corpus consists of Wall Street Journal documents from 1993-1994. The specific task was to detect management succession patterns. The authors also mention using a version of the MUC-6 corpus produced by Soderland (1999), which provided sentence-level specifications of succession patterns, requiring them to reconstruct the original text by aligning these patterns with sentences.

Key Insights and Ideas

  • Supervised Ranking is Key: The most significant insight is that integrating a supervised classifier (specifically, a maximum entropy classifier) into a semi-supervised IE framework dramatically improves accuracy. This allows the system to learn optimal feature weights automatically, a capability lacking in purely similarity-based approaches.
  • Active Learning for Negative Examples: The strategy of using low-similarity patterns as negative training examples is a clever way to efficiently teach the classifier what not to extract, thereby improving precision.
  • Richer Features Pay Off: The use of a more extensive feature set, beyond simple subject-verb-object triples, combined with the classifier's ability to assign weights, is a critical factor in achieving higher accuracy.
  • Bridging the Gap: This work demonstrates a viable path to making semi-supervised IE methods competitive with fully supervised or rule-based approaches in terms of accuracy, without the associated high user effort.

Key Results

The paper reports a significant improvement over previous weakly supervised IE methods. Their best result achieved an F-measure of 0.688 on the MUC-6 data set, which is a substantial increase from the 0.58 F-measure reported by Stevenson and Greenwood (2005) for a similar task. This improvement was attributed to the use of a supervised classification approach for ranking candidate patterns and the richer feature set employed. At the 36th iteration, the algorithm selected 180 sentences, correctly identifying 108 of the sentences containing succession patterns.

Limitations and Future Work

  • Human-in-the-Loop: The authors suggest evaluating the benefit of incorporating a human in the loop during the initial iterations to filter patterns chosen by the system.
  • Negative Sample Selection: They plan to investigate the use of random sub-selection of negative candidate patterns to better handle sentence fragments that might appear among low-ranking candidates.
  • Generalizability: While the MUC-6 dataset is used, further evaluation on other ground truth data sets is planned to verify the robustness of their approach.
  • Feature Contribution Analysis: The authors intend to detail how the various features in their classification model contribute to the ranking of candidate patterns.

Summary updated

BibTeX
@inproceedings{tratz2007high,
  author = {Stephen Tratz and Antonio Sanfilippo},
  title = {{A High Accuracy Method for Semi-Supervised Information Extraction}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2007},
}
Figure