TL;DR
This paper introduces "twin-systems" to tackle the explainability challenge in Artificial Intelligence (AI), specifically for "black box" Artificial Neural Networks (ANNs). By creating an interpretable "twin" model (Case-Based Reasoning, CBR) that mirrors the ANN's behavior, the authors demonstrate a novel way to understand ANN predictions. They also introduce and evaluate a new feature-weighting method, COLE, which proves most effective in this twinning approach.
What Problem Does This Paper Address?
As AI systems, particularly ANNs, become increasingly prevalent in decision-making across various domains, their lack of transparency poses a significant problem. Users and regulators alike are demanding explanations for AI-driven outcomes, a field known as Explainable AI (XAI). ANNs, with their complex, non-linear structures, are notoriously difficult to interpret, making it hard to understand why they make a particular prediction. This paper addresses this "black box" problem by proposing a "twin-system" approach, where a less interpretable ANN is paired with a more transparent CBR system. The goal is to leverage the interpretability of CBR to explain the ANN's decisions.
Methods
The core of the paper is the "twin-system" framework, specifically focusing on ANN-CBR twins. This involves:
- Twinning: Creating a CBR system that acts as a "twin" to an ANN. Both systems operate on the same dataset.
- Feature Weighting: Extracting feature weights from the ANN to inform the CBR system's reasoning process. This is crucial because the importance of features in explaining a prediction can vary.
- Novel Evaluation Methodology: A new method is proposed to quantitatively assess the "agreement" between the ANN and its CBR twin. High agreement signifies that the extracted feature weights accurately represent the ANN's decision-making process.
- COLE (Contributions Oriented Local Explanations): A novel feature-weighting method is introduced. It leverages saliency map techniques (like LRP, IG, and DeepLIFT) to estimate feature contributions, which are then used to weight the CBR's k-NN search. This method aims to capture how features directly contribute to the ANN's classification.
- Comparative Testing: The paper competitively tests several existing and the new COLE feature-weighting methods in two experiments.
Datasets
The paper utilizes several publicly available datasets for evaluation:
- Experiment 1 (MLP-CBR Twins):
- Connect-4
- Thyroid Disease
- Nursery
- Adult Census
- Default of Credit
- Bank Marketing
- Breast Cancer
- Experiment 2 (CNN-CBR Twins):
- MNIST
- MNIST-Fashion
- CIFAR-10
- CIFAR-10* (using transfer learning with InceptionResNetV2)
These datasets are chosen to challenge unweighted k-NN baselines and represent both tabular and image data.
Key Insights and Ideas
- Twin-Systems as a Promising XAI Approach: The paper strongly advocates for the twin-system framework as a viable method for achieving explainability, particularly by pairing ANNs with CBR systems.
- Feature Weighting is Key: The effectiveness of ANN-CBR twins hinges on accurately extracting feature weights from the ANN. Different weighting methods yield significantly different levels of agreement between the twin systems.
- Local vs. Global Weighting: Local feature-weighting methods generally outperform global methods, suggesting that feature importance is often context-dependent rather than a fixed, universal value.
- Contribution-Based Weighting Excels: The novel COLE method, which focuses on feature contributions, consistently achieves the highest agreement scores, indicating its superior ability to capture the ANN's decision logic.
- CNN Explanations are Plausible: For image data, the ANN-CBR twins, particularly with COLE, provide qualitatively convincing explanations, showing nearest neighbors that visually justify the CNN's classifications.
Key Results
- MLP-CBR Twins: The COLE method (specifically C-DeepLIFT) achieved the highest mean agreement score of 0.99, significantly outperforming other methods and the baseline unweighted k-NN (0.91). Local weighting methods generally showed better performance than global ones.
- CNN-CBR Twins: The C-DeepLIFT weighted k-NN achieved a very high mean agreement score of 0.996, demonstrating strong fidelity to the CNN's predictions. This performance was particularly notable in complex multi-class classification tasks.
- Qualitative Evidence: Figures in the paper illustrate how CNN-CBR twins can provide intuitive explanations for image classifications, showing nearest neighbors that visually resemble the queried image and highlighting key discriminating features.
Limitations and Future Work
- Focus on Agreement: The current evaluation methodology primarily focuses on predictive agreement between the twin systems, rather than directly assessing the "goodness" or user-understandability of the explanations themselves.
- Computational Cost: While the paper notes that deriving weights is a one-time cost and subsequent queries are fast, the initial computation for large datasets can still be significant (e.g., ~6613 seconds for CIFAR-10).
- Generalizability of COLE: The authors plan to generalize COLE to be applicable to machine learning models beyond just ANNs, such as Support Vector Machines (SVMs).
- User Studies: Future work will involve user studies to assess the explanatory value of the retrieved cases from the CBR twin.
- Broader Twin-System Applications: The paper suggests exploring other combinations of "black box" and "white box" models for twinning.
Summary updated