TL;DR
This paper introduces a novel method for generating "if-then-else" rules to globally explain the predictions of complex neural networks. By transforming input features based on their importance to the model and then applying rule induction, they achieve a high fidelity explanation, making black-box models more interpretable.
What Problem Does This Paper Address?
Deep neural networks, while powerful, are often criticized as "black boxes" because their decision-making processes are opaque. This lack of interpretability hinders trust, debugging, and understanding of potential biases. While many methods exist to identify important features, they often lose the crucial context of how these features interact to influence the final prediction. This paper aims to bridge this gap by providing a global explanation of the entire model's behavior through understandable rule sets.
Methods
The authors propose a three-step pipeline:
- Input Saliency Computation: They first calculate the "saliency" of each input feature for a given prediction. This is done by computing the gradient of the predicted output with respect to the input feature. A higher absolute gradient value indicates a more important feature. The sign of the gradient reveals whether the feature is positively or negatively correlated with the predicted class.
- Input Transformation and Selection: The original input features are then reweighted by their saliency scores. This transformed input space is further simplified by taking the sign of the reweighted values, resulting in discrete features representing positive correlation (1), negative correlation (-1), or no correlation (0). To manage complexity, they select the top 1000 features using either unsupervised sensitivity analysis or supervised mutual information between the transformed features and the training labels.
- Rule Induction: Finally, a rule induction algorithm (RIPPER-k) is applied to the transformed and selected features to learn if-then-else rules that mimic the original model's predictions. For multi-class problems, this is done in a one-vs-rest manner, generating separate rule sets for each class.
Datasets
The authors evaluate their technique on the 20-newsgroups dataset, specifically using documents from the 'Space', 'Medicine', 'Electronics', and 'Cryptography' categories. This subset was chosen to reduce experimental complexity, resulting in approximately 535 training, 60 development, and 395 test instances per category.
Key Insights and Ideas
- Feature Importance is Not Enough: Simply knowing which features are important doesn't tell us how they combine to make a decision. This work emphasizes capturing feature interactions.
- Transforming Inputs for Interpretability: Reweighting inputs by their model-specific importance before rule induction helps the rules better reflect the model's logic, not just general data patterns.
- Discrete Features for Readability: Simplifying transformed features to {-1, 0, 1} significantly enhances the readability and understandability of the induced rules.
- Elimination Strategy in Models: The induced rules reveal that models might use an "elimination strategy," where the absence of certain terms helps rule out other classes, in addition to identifying class-specific keywords.
- Generalization vs. Model Mimicry: The authors found that rules learned on the network-transformed data (mimicking the model's learned patterns) achieved a higher fidelity score than rules learned directly on the original data, suggesting that the model's generalization capabilities aid in rule induction.
Key Results
The proposed technique achieved a macro-averaged F-score of 0.80 when explaining the predictions of a 4-class text classifier trained on the 20-newsgroups dataset. The precision of the induced rules was high, indicating reliability. The F-scores varied by class, with 'Space' achieving the highest (0.90) and 'Electronics'/'Cryptography' the lowest (0.75), suggesting these latter classes are more confusable.
Limitations and Future Work
- Limitations:
- RIPPER-k's "else" clause for the majority class and its unsuitability for one-class problems or when the target class is the majority in a one-vs-rest setup.
- Difficulty in handling features that frequently co-occur, potentially leading to only one being captured.
- The technique provides global explanations, not explanations for individual instances.
- Learned rules are sensitive to RIPPER-k's hyperparameters, requiring optimization.
- Future Work:
- Evaluate the technique on datasets with varying complexities.
- Compare with other rule induction algorithms.
- Investigate rule induction algorithms that support one-class problems and are less sensitive to parameters.
Summary updated