Information Extraction Research and Applications: Current Progress and Future Directions

Andrew Kehler, Jerry R. Hobbs, Douglas E. Appelt, John Bear, Matthew Caywood, David J. Israel, Megumi Kameyama, David L. Martin, Claire MonteleoniTIPSTER 1998 · OpenReview

TL;DR

This paper surveys SRI International's progress in information extraction (IE) through the TIPSTER Phase III program, focusing on improving accuracy and portability. They introduce FASTUS, an IE system, and detail advancements in its pattern recognition, coreference resolution, and merging components, alongside efforts to make IE systems more adaptable to new domains.

What Problem Does This Paper Address?

The paper tackles two major hurdles in deploying information extraction (IE) technology: accuracy and portability. Current IE systems struggle to achieve the high levels of precision and recall needed for real-world applications, and building systems for new domains is a time-consuming, expertise-intensive process. The authors aim to push the boundaries of both by refining existing IE techniques and exploring new learning-based approaches.

Methods

The core of the paper revolves around SRI's FASTUS system. Key methodological advancements include:

  • Lattice-Based Pattern Recognition: Moving from finite-state transducers to a lattice-based approach allows for the representation of multiple analyses simultaneously, mitigating premature decisions based on insufficient local evidence. This helps in handling ambiguity more effectively.
  • Coreference Resolution Enhancements: The lattice system enables "delayed resolution," allowing coreference to be considered after domain-phase analysis. This addresses a catch-22 where coreference needs domain knowledge, and domain analysis needs coreference resolved. They also explore incorporating syntactic constraints for intrasentential coreference.
  • Fragment Analysis: A domain-specific technique to handle incomplete domain pattern matches by attempting to fill missing slots from the surrounding discourse context, similar to the merging phase but focused on individual events.
  • Learning Merging Strategies: Instead of hand-coding merging rules, the authors explore both supervised and unsupervised learning approaches.
  • Supervised: Using classification trees and maximum entropy models trained on annotated merge data.
  • Unsupervised: Employing Hierarchical Agglomerative Clustering (HAC) with custom similarity metrics, and an "indirectly supervised" approach that tunes similarity metrics based on end-to-end system performance.
  • Open Domain System: Development of a system for business news that uses a broad set of patterns and ontologies to enable the creation of "FASTLETS" (FASTUS-like systems) for specific information needs, aiming for broader applicability.
  • Application to Document Retrieval: Using FASTLETS to rerank documents, improving the quality of search results.

Datasets

  • MUC-6 Development Corpus: Used extensively for training and evaluating various components, including pattern recognition, coreference, and merging strategies.
  • MUC-6 Training Set (100 messages): Used for developing and testing learned merging strategies.
  • MUC-6 Test Set: Used for evaluating the overall system performance.
  • TIPSTER Phase III Annotated Training Data: An additional set of annotated data provided by the Office of Research and Development.
  • TREC-5 Corpus: Used in experiments for document retrieval applications.
  • TREC-6 Routing Task Data: FASTLETS were developed for 23 out of 47 topics in this task.
  • WSJ article 870112-0001: A specific example text used to illustrate issues with the MUC-6 scoring system.

Key Insights and Ideas

  • The "Catch-22" of Coreference: The paper highlights a fundamental challenge where coreference resolution often needs domain-level understanding, but domain analysis requires coreference to be resolved first. The lattice-based system offers a promising way to break this cycle.
  • Local vs. Global Evidence: While local evidence processing (as in FASTUS's initial design) is efficient, it can lead to premature, incorrect decisions. Lattice-based processing offers a way to retain multiple analyses and leverage more context.
  • The Mismatch Between Subcomponent Evaluation and End-to-End Performance: A significant finding is that improving a subcomponent (like merging) based on human annotations doesn't always translate to better overall system performance. This suggests that the scoring mechanism itself might be flawed or that the interaction between components is complex.
  • The "Zipf Effect" and Diminishing Returns: The paper questions the assumption that a fixed number of training messages (e.g., 100 in MUC-6) is sufficient to capture all relevant patterns. They suggest that even with minor fixes, improvements can transfer to unseen data, implying that the point of diminishing returns might be further out or dependent on data overlap.
  • The Limitations of Current Scoring Metrics: The authors express concern that the MUC-6 scoring mechanism may be misguiding efforts, as improvements that intuitively seem positive can lead to lower scores. This points to a critical need for better evaluation strategies.
  • The "What's on the Sleeve" Problem: The paper suggests that current IE approaches are good at extracting explicitly stated information but struggle with deeper, implicit meaning, requiring new and richer techniques for revolutionary improvements.

Key Results

  • Lattice-Based System: Experiments showed mixed results, with typical gains of about one point in F-score, generally increasing recall at the expense of precision.
  • Coreference Resolution: Both coreference and lattice components independently increased recall and precision. The combination led to a larger recall increase (13% error reduction) but less precision impact.
  • Fragment Analysis and Merging: Table 1 shows that fragment analysis alone improved F-score from 52.60% to 56.10%, and merging alone improved it to 57.67%. Combining both yielded the best result at 59.17%.
  • Open Domain System (FASTLETS): In TREC-6, FASTLETS were developed for 23 topics in an average of 4-6 hours per topic. The GE/SRI system improved DR results for 16 out of 47 topics, with significant improvements in 2 cases.
  • Trigger Word Compiler: Achieved a speedup of more than a factor of three for FASTUS processing.
  • Learning Merging Strategies: Indirectly supervised learning experiments showed a practically flat performance graph, with the best metrics achieving an F-score of 59.80%, but no significant improvement over the baseline.

Limitations and Future Work

  • Limitations:
    • The mixed results from lattice-based processing suggest that further work is needed to refine strategies for pruning and path selection.
  • The supervised learning experiments for merging did not significantly improve end-to-end performance, suggesting that the chosen features might not be sufficiently indicative of merge quality.
  • The MUC-6 scoring mechanism is identified as a potential limitation, as it may not accurately reflect true system improvements.
  • The paper acknowledges that current IE systems are good at "surface-level" information and need new techniques for deeper understanding.
  • Future Work:
    • Further experimentation with lattice-based systems and integration strategies.
  • Continued research into more effective evaluation strategies and scoring mechanisms.
  • Developing new and richer techniques for identifying implicit information in natural language.
  • Leveraging external knowledge bases and unsupervised mining of text corpora for improved IE.
  • Embedding IE technology into applications like document retrieval, summarization, and multimedia fusion.
  • Developing architectures that integrate DR and IE evidence more effectively.
  • Implementing learning algorithms for incorporating heterogeneous types of evidence.
  • Further development of the open domain infrastructure and user interfaces for rule adaptation.

Summary updated

BibTeX
@inproceedings{kehler1998information,
  author = {Andrew Kehler and Jerry R. Hobbs and Douglas E. Appelt and John Bear and Matthew Caywood and David J. Israel and Megumi Kameyama and David L. Martin and Claire Monteleoni},
  title = {{Information Extraction Research and Applications: Current Progress and Future Directions}},
  booktitle = {TIPSTER},
  year = {1998},
  url = {https://openreview.net/forum?id=OidhsHTxW1c},
}
Figure