A Data-Driven Knowledge Acquisition System: An End-to-End Knowledge Engineering Process for Generating Production Rules

Maqbool Ali, Rahman Ali, Wajahat Ali Khan, Soyeon Caren Han, Jae Hun Bang, Tae Ho Hur, Dohyeong Kim, Sungyoung Lee, Byeong Ho KangIEEE Access 2018 · OpenReview

Introduction

In the modern engineering landscape, data mining has become indispensable for extracting value from vast datasets. However, the transition from raw data to actionable knowledge remains a fragmented and complex process. Most existing methodologies focus on isolated stages—such as preprocessing or classification—rather than the entire lifecycle. This "siloed" approach creates a significant barrier for domain experts, such as healthcare professionals, who possess valuable data but lack the deep technical expertise required to navigate complex frameworks like the Cross-Industry Standard Process for Data Mining (CRISP-DM).

To bridge this gap, this research introduces the Data-Driven Knowledge Acquisition Tool (DDKAT). DDKAT is an end-to-end, web-based system designed to automate the journey from messy, raw datasets to human-readable IF-THEN production rules, allowing experts to focus on domain insights rather than technical implementation.

The DDKAT Framework

The DDKAT system is built on the Spring MVC framework and adheres to the CRISP-DM roadmap. It provides a unified pipeline that handles feature selection, data cleaning, and the generation of shareable, validated knowledge. The system is designed to be "human-in-the-loop," providing a user-friendly interface where experts can understand the rationale behind automated decisions.

Architecture of the DDKAT system within its multi-layered curation framework.

As illustrated in the architecture, the system operates within a Knowledge Curation Layer, supported by underlying data and information curation layers. The core Data-driven Knowledge Acquisition System orchestrates the flow from data sharing and preprocessing to algorithm selection and final rule generation.

Technical Innovations

The DDKAT approach introduces several key technical innovations to streamline the knowledge engineering process:

Unified Features Scoring (UFS)

Rather than relying on a single statistical metric, UFS employs an ensemble approach. It combines five filter-based measures: Information Gain, Gain Ratio, Symmetric Uncertainty, Chi-square, and Attribute Significance. These measures are normalized and aggregated to produce a "feature scorecard," allowing users to identify the most informative variables with high confidence. In testing, UFS matched or outperformed traditional single-measure rankings in 97.15% of cases.

Accurate Multi-criteria Decision Making (AMD)

To eliminate the "trial and error" typically associated with model selection, DDKAT utilizes a meta-learning framework. The AMD component evaluates dataset characteristics and recommends the optimal decision tree algorithm from a pool of seven candidates, including J48, BFTree, and RandomTree.

Automated Model Translation

A primary goal of DDKAT is to produce interpretable knowledge. While decision trees are "white-box" models, they can become unwieldy as they grow. The system employs a depth-first search (DFS) algorithm to traverse generated decision trees, translating every path into a simplified IF-THEN production rule. These rules are then exported via a RESTful JSON web service, making them easily consumable and easier for human experts to validate.

Experimental Validation

The system was validated using several benchmark datasets. The Pima Indians Diabetes Dataset served as the primary case study for end-to-end simulation. Additionally, seven datasets from the OpenML repository (including Sonar, Arrhythmia, Waveform, and Glass) were used to evaluate the UFS methodology and the impact of preprocessing.

Key Results

  • Preprocessing Impact: Experimental results confirmed a steady increase in classifier accuracy as data progressed through cleaning stages, specifically handling missing values, removing outliers, and discretization.
  • User Acceptance: A study involving 19 experts and 102 students rated the tool highly on the AttrakDiff scale. It received high marks for both "Pragmatic Quality" (task-related efficiency) and "Hedonic Quality" (innovativeness), officially categorizing it as a "desired" product.

Conclusion and Future Work

DDKAT represents a significant step toward democratizing data science. By providing an automated yet transparent framework, it allows domain experts to extract high-quality insights without needing specialized data science training.

Future research will address current limitations, including the memory intensity of the XML parser for very large trees and the need for robust user authentication. A major goal is the integration of "expert-driven" modules to automatically detect and flag conflicts where data-generated rules contradict established domain knowledge.

Summary updated

BibTeX
@inproceedings{ali2018data,
  author = {Maqbool Ali and Rahman Ali and Wajahat Ali Khan and Soyeon Caren Han and Jae Hun Bang and Tae Ho Hur and Dohyeong Kim and Sungyoung Lee and Byeong Ho Kang},
  title = {{A Data-Driven Knowledge Acquisition System: An End-to-End Knowledge Engineering Process for Generating Production Rules}},
  booktitle = {IEEE Access},
  year = {2018},
  url = {https://openreview.net/forum?id=NL4VtH5k5i},
}
Figure