Introduction
In the early 1990s, the field of Information Extraction (IE) was characterized by a stark divide between sentence-level parsing and discourse-level synthesis. While statistical methods were beginning to automate the extraction of entities from individual sentences, the task of "wrapping up" these fragments into a coherent, structured template remained a manual bottleneck. Discourse analysis—resolving coreferences, linking entities to events, and filtering spurious extractions—required thousands of lines of hand-coded "if-then" heuristics. These systems were notoriously brittle and expensive to port to new domains.
This paper presents Wrap-Up, a system designed to automate discourse-level knowledge acquisition. By utilizing inductive learning through ID3 decision trees, Wrap-Up demonstrates that the complex "common sense" logic required for discourse can be learned directly from a corpus of annotated examples, reducing development time from months to a single day of automated training.
Methodology: The Wrap-Up Architecture
Wrap-Up decomposes the discourse task into six discrete classification stages, each managed by a specialized ID3 decision tree. This modular approach allows the system to handle different types of linguistic and structural evidence independently.
- Slot Filter: Acts as a gatekeeper, determining if an extracted entity is relevant to the target domain or merely "noise."
- Slot Merge: Performs coreference resolution by deciding if two distinct mentions (e.g., "Mitsubishi" and "the company") refer to the same entity.
- Links: Establishes semantic relationships between different object types, such as linking a specific piece of equipment to its manufacturer or a manufacturing process.
- Links Merge: Handles complex scenarios where multiple objects might point to a single event, deciding whether to consolidate them or maintain them as separate entities.
- Orphans: A recovery mechanism that infers the existence of missing objects not explicitly named in the text but required by the domain logic.
- Slot Defaults: Fills in missing attributes using context-sensitive defaults learned from the training data.
The decision trees utilize a variety of features, including linguistic "triggers" (specific keywords or phrases), the relative distance between mentions in the text, and the inheritance of context from linked objects.
Evaluation and Results
The system was rigorously tested using the MUC-5 (Fifth Message Understanding Conference) micro-electronics domain. The training set comprised 800 documents with associated "answer key" templates. Wrap-Up was benchmarked against the Trainable Template Generator (TTG), a contemporary discourse component that still relied on significant manual heuristic coding.
As shown in the results, Wrap-Up consistently outperformed TTG across all primary metrics. On average, Wrap-Up achieved an F-measure of 36.8, compared to TTG’s 34.8. Beyond raw performance, the primary advantage of Wrap-Up was its efficiency; it reached this level of accuracy with zero manual programming, requiring only approximately one day of CPU time to process the training corpus.
Furthermore, the system demonstrated significant "noise tolerance." In tests where the sentence analyzer produced a high volume of false positives (e.g., extracting 404 company names when only 131 were correct), Wrap-Up’s filtering stage successfully discarded over half of the spurious entities while retaining 77% of the valid ones.
Conclusion and Future Work
Wrap-Up represents a shift in IE from rule-based engineering to corpus-driven learning. By proving that discourse logic can be induced from examples, the authors addressed one of the most significant barriers to portable IE systems. While the system's performance is inherently limited by the accuracy of the initial sentence-level extraction, the "Orphans" stage provides a novel method for mitigating parser failures. Future research directions include the implementation of feedback loops between the discourse and sentence layers to further improve extraction accuracy.
Summary updated