Advances of the DBLearn System for Knowledge Discovery in Large Databases

Jiawei Han, Yongjian Fu, Simon Tang

Abstract

DBMiner is a second-generation data mining system designed to extract sophisticated knowledge from large relational databases. Evolving from the DBLearn prototype, DBMiner leverages attribute-oriented induction (AOI) to generalize data into high-level concepts. This paper details the system's modular architecture, its tight integration with SQL servers, and its ability to perform diverse mining tasks—including characterization, association, and classification—through a user-friendly, query-based interface.

Introduction

In the mid-1990s, the rapid growth of industrial databases created a "data explosion," where the volume of stored information far outpaced the ability to analyze it. Traditional machine learning techniques often failed to scale to the size of these databases or lacked integration with standard SQL environments. DBMiner was developed to address these challenges by providing an efficient, scalable, and interactive tool for Knowledge Discovery in Databases (KDD).

System Architecture and Integration

The strength of DBMiner lies in its tight integration with relational database technology. Unlike standalone tools that require data export, DBMiner operates directly on SQL servers.

System architecture of DBMiner showing the integration between the GUI, SQL Server, and Discovery Module.

As shown in Figure 1, the architecture consists of a Graphical User Interface (GUI) that communicates with both an SQL Server and a specialized Discovery Module. This setup allows for "interactive knowledge mining," where users can refine their queries in real-time. A critical component is the Concept Hierarchy, which stores the rules for generalizing data (e.g., grouping specific ages into broader life stages). DBMiner automates the generation and adjustment of these hierarchies based on data distribution.

The Discovery Module

The core of the system's functionality resides in its modular Discovery Module.

The internal modular structure of the DBMiner Discovery Module, featuring specialized components for various mining tasks.

Figure 2 illustrates the various specialized sub-modules within DBMiner. These include:

  • Characterizer: Summarizes general features of a target class.
  • Discriminator: Compares features across different classes.
  • Association Rule Finder: Discovers "if-then" relationships between attributes.
  • Classifier: Provides class descriptions for predictive modeling.
  • Evolution and Deviation Evaluators: Analyze changes and anomalies over time.

Methodology: Attribute-Oriented Induction (AOI)

The primary engine for discovery is Attribute-Oriented Induction. AOI works by "climbing" the concept hierarchies to generalize specific data points into more abstract summaries. For instance, individual city names are generalized to provinces or countries. This process reduces the search space and makes patterns visible that would be obscured by raw, noisy data.

To make these complex tasks accessible, the authors introduced a high-level data mining query language. This language mimics standard SQL, allowing users to trigger mining tasks with familiar syntax, such as characterize "CS-Grants" from award A....

Evaluation and Results

DBMiner was evaluated using the NSERC Research Grant Database and various industrial datasets. The system demonstrated significant improvements over its predecessor, DBLearn, particularly in its functional breadth and performance. By utilizing set-oriented database operations, DBMiner achieved high efficiency in analyzing large-scale data. Users were able to express complex mining tasks in just a few lines of code, receiving results in the form of generalized relations, charts, and rules.

Conclusion and Future Work

DBMiner represents a major advancement in making data mining a practical tool for database administrators. However, several challenges remain, including the efficient handling of incremental updates as new data arrives and extending these techniques to non-relational data types like spatial or temporal databases. Future research will focus on better integrating discovered knowledge into existing systems to facilitate continuous learning.

Summary updated

BibTeX
@inproceedings{han1995advances,
  author = {Jiawei Han and Yongjian Fu and Simon Tang},
  title = {{Advances of the DBLearn System for Knowledge Discovery in Large Databases}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {1995},
}
Figure