Abstract
This paper presents a novel approach to improve Information Extraction (IE) by utilizing distant supervision to automatically learn indicative keywords for specific relations. By re-ranking text passages using these weighted keywords prior to deep linguistic analysis, the proposed IKFB method significantly enhances both the accuracy and efficiency of fact extraction from large-scale datasets. The researchers demonstrate that this hybrid pipeline outperforms standard search engines and traditional retrieval methods, providing a robust solution for the "needle in a haystack" problem in massive corpora.
Introduction
Extracting specific facts—such as identifying the employer of a particular individual—from massive document collections is a significant challenge. While traditional Information Extraction (IE) systems offer high accuracy through deep linguistic parsing, they are computationally expensive and slow. Conversely, standard Information Retrieval (IR) systems are efficient but lack the relational context necessary for precise extraction.
The authors identify passage retrieval as a critical bottleneck: if the initial search fails to prioritize relevant sentences, the subsequent IE phase is either inefficient or ineffective. The core challenge addressed is how to "teach" a search engine to recognize specific relationships without the need for extensive manual rule-writing or hand-labeled training data.
Methods
The researchers employ Distant Supervision to bridge the gap between IR and IE. This technique leverages existing knowledge bases to automatically generate training data. By taking known facts from Freebase (e.g., "Steve Jobs worked for Apple") and automatically labeling Wikipedia sentences containing these entities, the system learns to identify relation-specific signals.
An Indicative Score is calculated for every word based on its frequency in positive versus negative instances. This process yields thousands of weighted keywords—termed IKFB (Indicative Keywords Learned from Freebase)—for relations like employee_of and member_of.
These keywords are used to re-rank passages retrieved by a standard engine (Lemur), ensuring that sentences with high relational potential are processed first. The evaluation was conducted on the KBP 2010 corpus, comprising 1.8 million newswire and web documents.
Results and Key Insights
The IKFB method demonstrated superior performance across multiple metrics. For the employee_of relation, it achieved a Mean Reciprocal Rank (MRR) of 0.409, significantly outperforming the baseline (0.269) and the standard Lemur engine (0.180). For nearly a third of all queries, the very first passage retrieved contained the correct answer.
Integrating IKFB into an end-to-end pipeline—combining traditional IE with a shallow QA-style system—resulted in a 3–5% improvement in overall F1-score. This improvement primarily stems from recovering answers that traditional systems found too syntactically complex to parse.
Key insights from the study include:
- Non-Intuitive Keywords: Words like "currently," "until," and "since" proved highly indicative of employment relationships, often appearing in biographical snippets.
- Robustness of Scale: Automated generation of thousands of keywords provides a level of robustness against varied writing styles that manual rule-writing cannot match.
- Domain Adaptability: Despite being trained on Wikipedia, the system performed well on newswire and web data, suggesting consistent lexical signals for human relationships across different genres.
Conclusion
The study proves that distant supervision can effectively "teach" search engines to recognize complex relationships, significantly boosting the efficiency of Information Extraction. While the system shows some bias toward the training data (e.g., academic terms from Freebase), its ability to adapt across domains is a significant step forward. Future work will focus on refining the QA component and adapting the keyword learning process for noisier data sources like social media.
Summary updated