Abstract
A variety of information extraction techniques rely on the fact that instances of the same relation are "distributionally similar," in that they tend to appear in similar textual contexts. We demonstrate that extraction accuracy depends heavily on the accuracy of the language model utilized to estimate distributional similarity. An unsupervised model selection technique based on this observation is shown to reduce extraction and type-checking error by 26% over previous results, in experiments with Hidden Markov Models. The results suggest that optimizing statistical language models over unlabeled data is a promising direction for improving weakly supervised and unsupervised information extraction.
Introduction
Information Extraction (IE) is the process of automatically pulling structured facts—such as "Founded(Microsoft, Bill Gates)"—from raw text. A common approach to verifying these extractions is the distributional hypothesis, which posits that words with similar meanings appear in similar contexts. Systems use Assessment by Distributional Similarity (ADS) to compare the context of a new candidate extraction to known "seed" examples. However, accurately modeling these contexts is challenging, particularly with sparse data. This research investigates whether improving a language model’s general statistical accuracy (measured by perplexity) directly translates to better performance in specific IE tasks.
Methodology
The study compares two primary types of Statistical Language Models (SLMs):
- N-gram models: Traditional models that predict a word based on the $n-1$ preceding words, utilizing smoothing techniques like Modified Kneser-Ney.
- Hidden Markov Models (HMMs): Unsupervised models that represent sentences as sequences of hidden states.
The researchers introduced an innovation in how contexts are compared. Instead of relying on raw word counts, they used the language models to estimate the probability of a context. For HMMs, they found that comparing the hidden state distributions ($P(t|e)$) of extractions provided a significant performance boost. KL Divergence was employed to measure the distance between candidate contexts and seed contexts. To scale these models to 60 million tokens of web text, the team implemented a parallel HMM codebase using the Message Passing Interface (MPI) on a supercomputing cluster.
Evaluation Datasets
Performance was evaluated across three distinct datasets:
- Unary: Single-argument relations (e.g., identifying entities like Companies or Films).
- Binary: Two-argument "type-checking" tasks (e.g., verifying if arguments in "Conquered(X, Y)" are valid entity types).
- Wikipedia: A challenging dataset using "List of" pages to evaluate weakly supervised extraction across 36 relations.
Perplexity as a Proxy for Accuracy
A key finding of this work is the strong negative correlation (averaging -0.742) between a language model's perplexity and its extraction accuracy. Perplexity measures how "surprised" a model is by new text; lower perplexity indicates a better model. Because perplexity can be calculated using unlabeled data, it serves as an effective unsupervised proxy for selecting the best model for IE tasks.
Furthermore, the research suggests that latent states in HMMs are more informative than raw n-gram contexts. Even when an n-gram model achieved lower perplexity, HMMs often outperformed them in ADS because their hidden states better capture semantic "types."
Key Results
The implementation of automated optimization—selecting the HMM with the lowest perplexity—led to substantial gains. The best-performing configuration (an HMM with 100 hidden states) achieved a 26% reduction in error compared to previous benchmarks.
The correlation between lower perplexity and higher Area Under the Curve (AUC) remained consistent across various HMM configurations and datasets, validating the approach of using unlabeled data for "nearly-ceaseless automated optimization."
Conclusion
While the study demonstrates clear benefits, it also notes a "floor effect" on complex datasets like Wikipedia, where simple n-gram models struggled regardless of perplexity. This suggests that more complex representational power is required for diverse data. Future work could involve scaling to much larger models. While this research predates modern Transformer architectures, it establishes the theoretical foundation for why more powerful language models are essential for advancing unsupervised information extraction.
Summary updated