A Mutually Beneficial Integration of Data Mining and Information Extraction

AAAI 2000

TL;DR

This paper introduces DISCOTEX, a system that synergistically combines Information Extraction (IE) and Data Mining (DM) to improve both tasks. By using DM to discover predictive rules from extracted data, DISCOTEX enhances the recall of IE systems, demonstrating a powerful way to leverage unstructured text for knowledge discovery.

What Problem Does This Paper Address?

The paper tackles the challenge of extracting valuable knowledge from the ever-growing volume of unstructured text data available online. While Data Mining (DM) excels at finding patterns in structured databases, and Information Extraction (IE) can convert unstructured text into structured formats, these two fields have historically operated in isolation. The authors identify a gap: how can we effectively bridge these domains to unlock deeper insights from text? Specifically, they explore how IE can feed DM, and crucially, how DM can then feedback to improve the performance of IE, particularly in boosting its recall (finding more relevant information).

Methods

The core of DISCOTEX lies in its two-way integration of IE and DM.

  • Information Extraction (IE) Component: The system utilizes RAPIER, a machine-learning-based IE system that learns relational patterns to extract specific pieces of information (slot fillers) from natural language documents. RAPIER learns from user-annotated examples to build an extraction rule base.
  • Data Mining (DM) Component: After RAPIER extracts structured data from a corpus, standard DM techniques (specifically, C4.5RULES and RIPPER) are applied to this extracted database. The DM component learns "prediction rules" that describe relationships between different slot fillers. For instance, if a job posting mentions "Java" and "graphics," a DM rule might predict that "PhotoShop" is likely required.
  • Integration for Improved IE: The key innovation is using these mined prediction rules to enhance RAPIER's extraction process. When RAPIER initially extracts information from a document, DISCOTEX then uses the prediction rules to suggest additional slot fillers that might have been missed. These predicted fillers are only extracted if they are confirmed by a substring match in the original document, thus aiming to improve recall without introducing too much noise.

Datasets

The primary dataset used for training and evaluation is a corpus of 600 user-annotated computer science job postings from the USENET newsgroup austin.jobs. For the text mining phase, an additional 4,000 unannotated documents from the same newsgroup were optionally used to build a larger database for mining prediction rules.

Key Insights and Ideas

  • Mutual Benefit: The paper strongly advocates for the synergistic relationship between IE and DM. IE provides the structured data necessary for DM, while DM can discover predictive relationships that significantly improve IE's performance.
  • Boosting Recall: A major finding is that DM can effectively boost the recall of IE systems. By predicting missing slot fillers based on learned relationships, DISCOTEX can identify information that the initial IE system might have overlooked.
  • Confirmation is Key: The strategy of confirming predicted fillers via substring matching in the original document is crucial. This prevents the system from hallucinating information and helps maintain a reasonable level of precision while improving recall.
  • Value of Unlabeled Data: While labeled data is essential for training the initial IE system, unlabeled data can be valuable for the DM component. Using more unlabeled data to build a larger database for rule mining leads to improved recall and F-measure.

Key Results

Experiments on the computer science job posting domain showed that DISCOTEX, by integrating mined prediction rules, achieved:

  • Improved Recall: DISCOTEX consistently outperformed RAPIER alone in recall, with the advantage increasing as more training data was used.
  • Moderate F-measure Increase: While precision saw a slight decrease, the overall F-measure (a balance of precision and recall) was moderately increased, indicating a net improvement in extraction performance.
  • Impact of Unlabeled Data: Using up to 4,000 unlabeled documents for rule mining led to recall rates up to 83.3% and F-measures up to 88.1%, compared to 77.6% recall and 86.4% F-measure for RAPIER alone on the same labeled data.
  • Effectiveness of Rule Mining: A strawman approach of simply extracting all known filler strings performed significantly worse than DISCOTEX, highlighting the value of learned predictive rules over simple heuristics.

Limitations and Future Work

  • Manual Slot Filler Collapsing: The current system requires manual effort to collapse similar slot fillers (e.g., "WinNT," "Windows NT") into canonical forms. Automating this through clustering or other similarity metrics is suggested.
  • Discrete-Valued Slots: The current approach primarily focuses on discrete-valued slots. Extending the rule miner to handle real-valued slots (like "desired years of experience") using regression techniques is a promising direction.
  • Automated Slot Selection: The selection of slots for rule mining was manual. Automating this process by identifying and quantifying correlations between slot values is proposed.
  • Improving Precision: While recall is improved, the paper acknowledges that methods for using discovered rules to improve precision are also needed. Simply discarding unpredicted fillers might damage recall. Combining confidence measures from both extraction and prediction rules is suggested as a way forward.
  • Broader Domain Evaluation: The authors plan to evaluate DISCOTEX on other domains like business news, medical records, and product order emails to assess its generalizability.

Summary updated

BibTeX
@inproceedings{2000mutually,
  title = {{A Mutually Beneficial Integration of Data Mining and Information Extraction}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2000},
}
Figure