Rudy Setiono and Huan

IJCAI 1995

TL;DR

This paper introduces a novel three-phase algorithm to extract human-readable rules from trained neural networks, bridging the gap between their high predictive accuracy and their notorious "black box" nature. By pruning networks and discretizing hidden unit activations, the authors demonstrate that neural network predictions can be as interpretable as those from decision trees, without sacrificing accuracy.

What Problem Does This Paper Address?

Neural networks, particularly those trained with backpropagation, often achieve superior predictive accuracy compared to traditional decision trees in classification tasks. However, their internal workings are opaque, making it difficult to understand why a particular prediction is made. This lack of interpretability, often termed the "black box" problem, hinders their adoption in domains where explanations are crucial, such as medical diagnosis or financial decision-making. While decision trees naturally produce interpretable rules, neural networks learn complex, non-linear relationships through weighted connections that are hard to translate into human-understandable logic. This paper aims to bridge this interpretability gap by developing a method to extract rules from neural networks that are both accurate and comprehensible.

Methods

The authors propose a "three-phase algorithm" to achieve rule extraction:

  1. Weight-Decay Backpropagation: The neural network is trained using backpropagation, but with an added weight decay penalty. This encourages smaller weights, effectively highlighting more important connections and making the network more amenable to pruning.
  2. Network Pruning: After training, the network is pruned to remove insignificant connections. This process is iterative, aiming to reduce the network's complexity while maintaining its predictive accuracy. The algorithm described involves removing weights below a certain threshold and retraining. If accuracy drops below an acceptable level, the process stops.
  3. Rule Extraction via Discretization: Once the network is pruned, the continuous activation values of the hidden units are discretized. This is achieved through a clustering approach. By grouping similar activation values and representing them with a single value, the continuous nature of hidden units is converted into discrete states. Rules are then extracted by examining the relationships between these discrete hidden unit activations and the network's output. The paper mentions using an algorithm that generates 100% accurate rules from these discretized activations. For particularly complex subnetworks, a recursive approach of creating smaller subnetworks for rule extraction is also described.

Datasets

The paper evaluates its method on three well-known benchmark datasets:

  • Iris: A classic dataset with 150 examples of Iris flowers, described by four numeric attributes.
  • Breast Cancer: A widely used dataset for breast cancer diagnosis, with 699 examples.
  • Splice-junction: A dataset used for splice-junction determination in DNA sequences, containing 3175 examples.

The authors note that for the Iris dataset, a ChiMerge algorithm was used to discretize attribute values, and a thermometer code was applied for input representation.

Key Insights and Ideas

  • Bridging the Interpretability Gap: The core idea is that by carefully pruning and discretizing a neural network, its complex internal logic can be translated into a set of understandable rules, similar to those generated by decision trees.
  • Pruning for Simplicity: Weight decay and subsequent pruning are crucial steps that reduce network complexity without significantly impacting accuracy, making rule extraction feasible.
  • Discretization as a Key Step: Converting continuous hidden unit activations into discrete states is essential for generating symbolic rules. The paper shows that this can be done with minimal loss of accuracy.
  • Rules Reflect Network Behavior: The extracted rules are presented as a direct mapping of the pruned network's behavior, meaning they accurately represent how the network makes predictions.
  • Comparable to Decision Trees: The paper strongly argues that the extracted neural network rules are comparable to, and in some aspects better than, rules generated by traditional decision tree algorithms (like C4.5rules) in terms of accuracy, conciseness, and interpretability.

Key Results

  • High Predictive Accuracy: The extracted rules preserve the high predictive accuracy of the pruned neural networks. For the Iris dataset, the pruned network achieved 97.33% accuracy on the test set, and the extracted rules maintained this level.
  • Concise Rules: The number of rules extracted from neural networks was often less than those generated by decision trees, contributing to better understandability.
  • Comparable Rule Complexity: The average number of conditions per rule was comparable between neural network rules and decision tree rules, with the paper noting that NN rules can be more complex but are still understandable.
  • Robustness: The algorithm demonstrated robustness, with experiments involving 100 pruned networks showing impressive average performance.

Limitations and Future Work

  • Training Time: The paper acknowledges that training neural networks is significantly more time-consuming than training decision trees.
  • Nondeterministic Nature: Due to the nondeterministic nature of neural network training (initial weights, etc.), multiple runs are often needed to achieve optimal results, as demonstrated by the 100-run experiment for the Iris dataset.
  • Rule Complexity vs. Accuracy Trade-off: While the algorithm aims to maintain accuracy, there's an inherent trade-off: further pruning for simpler rules might slightly reduce accuracy.
  • Future Work: The paper suggests that the ability to extract rules that reflect network behavior and maintain accuracy means neural networks should no longer be considered purely "black boxes." The authors imply that this method opens doors for more widespread adoption of neural networks in interpretable AI applications.

Summary updated

BibTeX
@inproceedings{1995rudy,
  title = {{Rudy Setiono and Huan}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {1995},
}
Figure