TL;DR
This paper introduces EXDISCO, an unsupervised method for automatically discovering patterns used in information extraction (IE) systems. By starting with a few "seed" patterns, EXDISCO iteratively expands its knowledge base, significantly reducing the manual effort traditionally required to adapt IE systems to new domains or event types. This approach achieves performance comparable to manually tuned systems, demonstrating a path towards more portable and efficient IE.
What Problem Does This Paper Address?
Information Extraction (IE) systems are designed to pull specific pieces of information from unstructured text, such as identifying corporate executives starting or leaving positions. A major hurdle in deploying these systems to new domains or "scenarios" (specific types of events or relations) is identifying the vast array of linguistic expressions used to describe these events. Traditionally, this involves extensive manual analysis and annotation of large text corpora, a time-consuming and expensive process that hinders the portability and widespread adoption of IE technology. The paper aims to automate this discovery process, reducing the reliance on human experts and large-scale manual annotation.
Methods
The core of the paper is the EXDISCO (Extraction Discovery) procedure. It's an unsupervised approach that starts with a small set of user-provided "seed" patterns. EXDISCO then iteratively performs the following steps:
- Corpus Partitioning: The entire corpus is divided into "relevant" and "non-relevant" documents based on whether they contain any of the current set of patterns.
- Candidate Pattern Generation: Each document is converted into potential "candidate" patterns, typically derived from its syntactic structure (e.g., verb-object pairs, subject-object pairs).
- Pattern Ranking: Candidate patterns are ranked based on how strongly their distribution correlates with document relevance. Patterns appearing more frequently in relevant documents than in non-relevant ones are favored.
- Pattern Set Expansion: The highest-ranking pattern is added to the set of known patterns.
- Document Re-ranking: Document relevance scores are updated based on the expanded pattern set. This process repeats until a stopping criterion is met (e.g., iteration limit or no new patterns found).
The system also incorporates syntactic analysis (using a dependency parser) and normalization steps to handle variations like passive voice and to replace named entities with their classes (e.g., C-Person, C-Company).
Datasets
The experiments were conducted using a corpus of 9,224 articles from the Wall Street Journal. For evaluation, they used the MUC-6 formal training corpus (100 documents) and the MUC-6 formal test corpus (100 documents). They also tested on an "Acquisition" task using a set of 200 documents retrieved via keyword search and manually judged for relevance.
Key Insights and Ideas
- Circularity Solved: The paper tackles the inherent circularity in pattern discovery: you need relevant documents to find patterns, but you need patterns to identify relevant documents. EXDISCO resolves this by acquiring patterns and relevant passages in tandem.
- Unsupervised Domain Knowledge Acquisition: The ability to automatically discover domain-specific patterns without requiring annotated data or manual relevance judgments is a significant step towards making IE systems more adaptable.
- Pattern-Based Document Filtering: EXDISCO's iterative process not only discovers patterns but also generates relevance rankings for documents, which can be used for text filtering tasks.
- Performance Beyond Manual Tuning: The results suggest that an automatically discovered set of patterns can outperform a system tuned manually for a month by an expert, highlighting the efficiency gains.
Key Results
- Management Succession Scenario:
- The "Seed" patterns (initial set) achieved a recall of 38% and precision of 83% (F=52.60) on the training corpus.
- The EXDISCO-discovered patterns, when added to the seed set ("Union"), achieved a recall of 69% and precision of 79% (F=73.50).
- This performance was comparable to or better than manually developed systems (Manual-MUC: F=61.93, Manual-NOW: F=73.91).
- On the test corpus, the EXDISCO-based system achieved an F-score of 60.16, while the manually developed Manual-NOW achieved 64.04.
- Text Filtering: Figures 1 and 2 show that EXDISCO's document relevance rankings achieve competitive recall-precision trade-offs, falling within the range of human-supervised MUC participants.
Limitations and Future Work
- Manual Review and Augmentation: While EXDISCO is largely automatic, the discovered patterns were reviewed and augmented manually to map them to predicate arguments for evaluation. This step, though brief, means the evaluation is not entirely automatic.
- Limited Pattern Types: The current system focuses on clause-level patterns. Some relevant information might be expressed in noun phrases or require more complex inferencing, which EXDISCO doesn't fully capture yet.
- Future Directions:
- Using larger training corpora to find less frequent patterns and potentially exceed the performance of hand-trained systems.
- Capturing word classes generated during discovery to improve pattern generalization and find less frequent patterns in subsequent iterations.
- Evaluating EXDISCO on a wider variety of scenarios to assess its general applicability.
- Understanding how the discovered topic clusters align with pre-specified scenarios.
Summary updated