ICE: Rapid Information Extraction Customization for NLP Novices

Yifan He, Ralph GrishmanNAACL 2015

TL;DR

This paper introduces ICE, an Integrated Customization Environment designed to empower non-NLP experts to rapidly build custom Information Extraction (IE) systems. By abstracting away complex linguistic formalisms and guiding users through an interactive process, ICE significantly lowers the barrier to entry for IE system development.

What Problem Does This Paper Address?

Creating effective Information Extraction (IE) systems, which identify and categorize specific pieces of information (like entities and their relationships) within unstructured text, is a notoriously difficult and time-consuming task. It typically requires deep linguistic expertise, extensive training, and significant computational resources. This complexity has hindered the widespread adoption of IE technologies. Existing tools often assume a high level of user proficiency, demanding familiarity with system internals and formal notations, leaving a gap for domain experts who understand the data but lack NLP backgrounds.

Methods

ICE tackles this challenge by providing a guided, interactive environment that abstracts away much of the underlying complexity. The system operates in five key steps:

  1. Preprocessing: A new corpus is first processed using a deep linguistic pipeline (Jet NLP) to generate caches of POS tags, noun chunks, dependency relations, named entities, and coreference chains. This one-time cost enables subsequent steps to run very quickly.
  2. Key Phrase Extraction: ICE identifies important terms and multi-word phrases by ranking them based on their frequency within the target domain corpus relative to a general corpus. This helps highlight domain-specific vocabulary.
  3. Entity Set Construction: Users define new entity types by providing "seeds" – representative examples of entities. ICE then uses distributional similarity based on dependency analysis to suggest and expand these seed sets, allowing users to accept or reject suggested terms.
  4. Dependency Path Extraction and Linearization: ICE captures semantic relations by identifying lexicalized dependency paths (LDPs) between entities. To make these paths more user-friendly, they are "linearized" by retaining only the essential words from the sentence, creating more readable representations.
  5. Bootstrapping Relation Extractors: ICE builds relation extractors using both exact and fuzzy matching of linearized LDPs against user-provided examples. It employs a bootstrapping approach, similar to Snowball, to iteratively refine positive and negative rules based on user feedback, allowing for the discovery of new patterns.

Datasets

The paper describes experiments conducted on a corpus of approximately 5,000 web news posts from the U.S. Drug Enforcement Administration (DEA). For evaluation, they manually annotated 51 documents from this collection, identifying SELL and RESIDENT-OF relations, and used these as a "gold standard" for assessing the performance of ICE-built systems.

Key Insights and Ideas

  • Bridging the NLP Divide: The core insight is that by abstracting linguistic formalisms and providing a guided, interactive interface, NLP novices can effectively build sophisticated IE systems.
  • Distributional Similarity for Entity Expansion: Leveraging dependency analysis to compute distributional similarity is a powerful way to automatically expand seed entity sets, significantly reducing manual effort.
  • Linearized Dependency Paths (LDPs) for Readability: Presenting linearized LDPs instead of raw dependency paths makes the system much more accessible to non-experts, facilitating their understanding and interaction.
  • Fuzzy Matching's Importance for Recall: For relations with diverse linguistic expressions (like "SELL"), fuzzy matching is crucial for achieving reasonable recall, even at the cost of some precision. Conversely, for more constrained relations ("RESIDENT-OF"), exact matching can be more effective.
  • Bootstrapping with Negative Rules: The inclusion of negative rules in the bootstrapping process helps to significantly reduce false positives, particularly for fuzzy matching extractors.

Key Results

The experiments demonstrated the utility of ICE:

  • Entity Set Construction: ICE achieved high-recall dictionary-based entity taggers, reaching 74% for DRUGS and 82% for AGENTS in just a few minutes of user interaction.
  • Relation Extraction:
    • With small rule sets, the fuzzy extractor (with negative rules) achieved F-scores of 0.32 for SELL and 0.58 for RESIDENT-OF.
  • With larger rule sets, the F-score improved to 0.40 for SELL and 0.62 for RESIDENT-OF.
  • The fuzzy match extractor generally outperformed exact match, especially for the SELL relation, highlighting the benefit of fuzzy matching for recall.
  • For RESIDENT-OF, the exact match extractor performed better, suggesting that for more constrained relations, precision can be prioritized.

Limitations and Future Work

  • Limitations: The paper acknowledges that the current system relies on dependency paths, which might not be expressive enough for all types of relations, particularly those involving multiple arguments or complex event structures. The fuzzy matcher can be prone to false positives, especially with large rule sets.
  • Future Work: The authors plan to explore more expressive models beyond LDPs that can handle an arbitrary number of arguments. This would enable ICE to build event extractors, further expanding its capabilities.

Summary updated

BibTeX
@inproceedings{he2015ice,
  author = {Yifan He and Ralph Grishman},
  title = {{ICE: Rapid Information Extraction Customization for NLP Novices}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2015},
}
Figure