TL;DR
This paper tackles the inherent noise and imprecision in Information Extraction (IE) pipelines by proposing a novel "redundancy-based correction" approach. Instead of relying solely on local, single-document analysis, it leverages the aggregated knowledge from a large collection of extracted facts to identify and correct errors, achieving significant improvements without requiring supervised training.
What Problem Does This Paper Address?
Information Extraction (IE) systems aim to pull structured facts from unstructured text, but they operate in a sequential "pipeline" of stages. Each stage, from basic tokenization to complex event matching, can introduce errors. These errors compound as they move through the pipeline, leading to a significant degradation in overall accuracy. Traditional IE systems typically propagate only the "best guess" hypothesis at each stage and process documents in isolation, making it difficult to recover from these accumulated errors. This paper addresses the challenge of improving IE accuracy by exploiting the redundancy present across a large corpus of extracted facts, treating this aggregated "a posteriori" knowledge as a valuable resource for error correction.
Methods
The core idea is to use global analysis of extracted facts to correct local errors. The authors propose several methods, starting with a Baseline: Raw Majority approach. This method identifies "ambiguous" location names (those that map to multiple states in the extracted data) and suggests the most frequent state associated with that location across the entire database as the correction.
Building on this, they introduce Database Filtering, which refines the majority vote by considering only states mentioned within the same story as the incident, aiming to filter out irrelevant majority winners.
A more significant improvement comes from Confidence-Based Ranking. Here, the system assigns a confidence score to each extracted state-location association based on heuristics related to the text span and uniqueness of the mention within the document. The majority vote is then weighted by these confidence scores.
Finally, Multi-Candidate Propagation extends the idea by allowing multiple candidate state hypotheses to be propagated through the system. The scoring function in this method considers the "proximity" of candidate states to the location mention across different stories, effectively sharing the "point" for a correct association among plausible candidates.
Datasets
The experiments were conducted on a large corpus of facts extracted from the ProMED-Mail mailing list. This list contains reports about outbreaks of infectious epidemics. The data spans over 10 years and consists of approximately 46,317 individual facts/incidents extracted from 30,015 stories within 22,560 reports. The authors note that they used a relatively small gazetteer for geographic knowledge to make the experiments more challenging and focused on the redundancy-based correction aspect.
Key Insights and Ideas
- Global redundancy is a powerful source of error correction: The paper strongly suggests that aggregating information across a large corpus of extracted facts can significantly improve the accuracy of individual extractions, even when the underlying IE process is noisy.
- Moving beyond the "best guess": Propagating multiple, locally non-best slot fills (hypotheses) through the pipeline, as done in the multi-candidate approach, unlocks valuable information for global analysis.
- The "state" field is a good testbed for global correction: The authors chose to focus on the location-state relation due to its relative simplicity and the presence of ambiguity, making it a suitable starting point for exploring redundancy-based correction.
- No supervised training needed: A key advantage of their approach is that it achieves substantial improvements without requiring manually annotated data for training, relying instead on the inherent redundancy in the extracted data.
Key Results
The paper demonstrates that their redundancy-based correction methods yield substantial gains over the baseline. The Baseline: Raw Majority method, while simple, achieved a "gain" of 68.5% (incorrect states changed to correct ones) with a "loss" of only 6.3% (correct states changed to incorrect ones). The Confidence-Based Ranking method further improved the gain to 76.4% while reducing loss to 3.5%. The Multi-Candidate Propagation method showed the best performance, achieving a gain of 76.4% and a loss of 10.9%, while changing a significantly larger number of records (16.2% vs. 13.0% for the baseline). This indicates that their methods are effective at identifying and correcting errors, particularly in the location-state relation.
Limitations and Future Work
The authors acknowledge that their current work focuses on a single relation (location-state) and that the geographic knowledge base used was intentionally modest. They suggest several avenues for future work:
- Combining evidence from multiple features: Beyond just distance, incorporating other textual features (e.g., mention order, headline presence) to improve scoring.
- Applying redundancy-based correction to other fields: Extending the methods to other key fields like disease name and date.
- Back-propagation of corrected results: Investigating how corrected facts can be used to repair components that induced incorrect information earlier in the pipeline.
- Leveraging high-confidence records for training: Using records with high confidence scores as a form of "ground truth" to train parameters for more complex models involving multiple features.
Summary updated