NeuroRule: A Connectionist Approach to Data Mining

Hongjun Lu, Rudy Setiono, Huan Liu

Abstract

Neural networks are often criticized as "black boxes" due to their lack of interpretability, making them difficult to use in data mining applications where symbolic rules are preferred. This paper introduces NeuroRule, a three-stage pipeline that transforms trained neural networks into concise, human-readable symbolic rules. By combining network training with weight decay, systematic pruning, and activation discretization, NeuroRule extracts classification rules that match or exceed the accuracy of traditional decision tree algorithms like C4.5 while maintaining greater conciseness.

Introduction

In the mid-1990s, the field of data mining was split between symbolic learning methods, such as decision trees, and connectionist approaches, primarily neural networks. While neural networks (NNs) demonstrated high predictive accuracy and robustness to noise, they suffered from the "articulating problem." Knowledge in an NN is distributed across a complex web of weights and nodes, making it nearly impossible for humans to understand the logic behind a specific classification. Data mining, however, requires explicit rules—such as "If salary > $50k and age < 40, then Group A"—to be useful for expert verification or database integration. NeuroRule bridges this gap by extracting interpretable rules from neural networks without sacrificing their inherent performance.

The NeuroRule Pipeline

The authors propose a three-step process to convert a standard feedforward neural network into a set of symbolic rules:

  1. Network Training: A standard three-layer feedforward neural network is trained using the BFGS algorithm, a quasi-Newton method that offers faster convergence than standard backpropagation. A critical component of this stage is the inclusion of a penalty function (weight decay) in the error calculation. This encourages the network to keep weights small, facilitating the subsequent pruning stage.
  2. Network Pruning (NP Algorithm): A fully connected network is typically too complex for direct rule extraction. The NP algorithm systematically removes redundant links and nodes. If the removal of a specific weight does not significantly degrade model accuracy, it is eliminated. This results in a "skinny" or sparse network architecture where only the most vital connections remain.
A pruned neural network showing positive and negative weights, illustrating the result of the NP algorithm.
  1. Rule Extraction (RX Algorithm): The core innovation of NeuroRule lies in how it handles hidden unit activations. In a pruned network, these activations tend to cluster into distinct values. The RX algorithm discretizes these activations using clustering techniques, generates rules mapping inputs to these discrete hidden states, and then maps those states to the final output. These intermediate steps are then collapsed into final "if-then" rules through substitution.

Experimental Setup

NeuroRule was evaluated using a benchmark suite of ten synthetic classification functions originally proposed by Agrawal et al. (1992). These functions simulate real-world entities with nine distinct attributes.

The training sets consisted of 1,000 tuples, while test sets ranged from 1,000 to 10,000 tuples. The performance of NeuroRule was compared against C4.5, the industry-standard symbolic decision tree algorithm of the era.

Key Insights and Results

The study yielded several significant findings regarding the connectionist approach to rule extraction:

  • Pruning is Essential: The authors emphasize that pruning is not merely an optimization for efficiency but a prerequisite for interpretability. Clean rules cannot be extracted from dense, fully connected networks.
  • Hidden Units as Logical Concepts: By discretizing hidden unit outputs, the RX algorithm effectively treats these units as intermediate logical gates, transforming continuous mathematical functions into discrete logic.
  • Conciseness and Accuracy: NeuroRule's accuracy was comparable to, and in some cases superior to, C4.5. For instance, on "Function 2," NeuroRule achieved 100% test accuracy, whereas C4.5 reached 96%. More importantly, NeuroRule produced significantly more compact rule sets. For the same function, C4.5 generated 18 rules, while NeuroRule extracted just 4 rules that perfectly captured the underlying logic.
  • Robust Attribute Selection: NeuroRule proved more effective at ignoring irrelevant "noise" attributes. While C4.5 occasionally included irrelevant features like car make in its rules, NeuroRule correctly identified only the relevant attributes.

Conclusion

NeuroRule demonstrates that neural networks can be more than just "black boxes." By applying rigorous pruning and discretization, it is possible to extract symbolic rules that are both accurate and concise. While the authors acknowledge that training and pruning are computationally more expensive than building decision trees, they frame NeuroRule as a successful "first attempt" at applying connectionist methods to data mining. Future work will focus on improving scalability for high-dimensional data and reducing overall training time.

Summary updated

BibTeX
@inproceedings{lu1995neurorule,
  author = {Hongjun Lu and Rudy Setiono and Huan Liu},
  title = {{NeuroRule: A Connectionist Approach to Data Mining}},
  booktitle = {VLDB},
  year = {1995},
}
Figure