TL;DR
This paper introduces "Sensitivity Consistency" (SenC), a novel black-box evaluation metric for explainability methods in deep neural networks. SenC assesses how consistently predictions and their explanations react to changes in input features and model parameters, aiming to provide a more robust measure of explanation reliability.
What Problem Does This Paper Address?
Deep neural networks (DNNs) are increasingly powerful but often act as black boxes, making it hard to trust their decisions, especially in critical applications like autonomous driving or healthcare. Explainability methods aim to shed light on DNNs by highlighting important input features or model components. However, evaluating these methods is challenging due to the lack of ground truth. Existing evaluation metrics often focus on a single aspect (e.g., fidelity, robustness), leading to an incomplete picture of an explanation's quality. This paper addresses this gap by proposing a new evaluation perspective that considers the consistency of how both the model's predictions and its explanations respond to perturbations. The intuition is that if a feature strongly influences a prediction, it should also strongly influence the explanation of that prediction, and vice-versa.
Methods
The core contribution is the Sensitivity Consistency (SenC) metric, which operates as a black-box approach, meaning it doesn't require access to the model's internal architecture or gradients. SenC has two main components:
- Data Sensitivity Consistency: This component assesses how consistently predictions and explanations react to perturbations in input features. It works by:
- Generating a large number of random masks to perturb (e.g., remove or alter) parts of the input data.
- Predicting the output of the original and perturbed inputs to measure the change in prediction confidence ($\Delta P$).
- Generating explanations for both the original and perturbed inputs using the explainability method under scrutiny.
- Calculating "prediction sensitivity" based on the weighted sum of $\Delta P$ across all masks, and "explanation sensitivity" based on the similarity of explanations between original and perturbed inputs.
- Finally, SenC is computed as the correlation (specifically, Spearman's rank correlation) between these prediction and explanation sensitivities. A high correlation indicates good sensitivity consistency.
- Parameter Sensitivity Consistency: This extends the concept to model parameters. It involves perturbing groups of parameters within specific layers of the network and observing how both predictions and explanations change. Similar to data sensitivity, the correlation between prediction and explanation sensitivity to parameter changes is calculated. This component is more computationally intensive and requires careful consideration of how to group parameters for perturbation.
The paper also introduces Top-K agreement metrics to focus on the most sensitive features.
Datasets
The paper evaluates SenC on several standard datasets:
- MNIST: A simple, low-complexity dataset for initial evaluation.
- CIFAR-10: A more complex image dataset.
- GTSRB (German Traffic Sign Recognition Benchmark): A real-world dataset.
- ImageNet: Used for a user study with a large, complex model (ViT).
Key Insights and Ideas
- Consistency as a Proxy for Reliability: The core idea that explanations should mirror the model's own sensitivity to input features and parameters is a powerful new lens for evaluation. If an explanation highlights a feature that the model largely ignores, it raises questions about the explanation's validity.
- Black-Box Nature is Key: SenC's black-box nature makes it broadly applicable to any explainability method, a significant advantage over gradient-based or model-specific metrics.
- Data vs. Parameter Sensitivity: The paper highlights that sensitivity consistency can be evaluated at both the input data level and the model parameter level, offering a more comprehensive assessment.
- Layer-wise Parameter Sensitivity: Parameter sensitivity consistency varies significantly across different layers of a neural network, with convolutional layers generally showing higher consistency than fully-connected layers. This suggests that explanations might be more reliable for features directly extracted from raw input rather than abstract, high-level representations.
- User Study Alignment: The SenC metric shows a promising correlation with human intuition and user ratings, suggesting it captures aspects of explanation quality that humans perceive.
Key Results
- Quantitative Evaluation: SenC was applied to several popular explainability methods (VB, GB, IG, LRP, GradCAM, DeepLift) across different datasets and models. Results generally showed that methods like IG and DeepLift tended to exhibit higher sensitivity consistency, especially on more complex datasets.
- Dataset Complexity Matters: Sensitivity consistency generally decreases as dataset and model complexity increases. For instance, LRP's consistency dropped significantly from MNIST to CIFAR-10.
- Parameter Sensitivity Trends: Parameter sensitivity consistency was found to be higher in convolutional layers compared to fully-connected layers across all datasets.
- User Study: On ImageNet with a ViT model, methods like IG and GradCAM received higher user scores and also showed better SenC, indicating a correlation between the metric and human perception of explanation quality.
Limitations and Future Work
- OOD Perturbations: The paper acknowledges that using random masks for perturbation constitutes "hard perturbation," which can disrupt data distribution and potentially affect evaluation reliability. Alternatives exist but are computationally intensive.
- FC Layer Perturbation: Perturbing fully-connected (FC) layers is challenging due to their dense connectivity and the sheer number of parameters. Finding an efficient and effective way to segment and perturb FC layers without sacrificing accuracy or computational feasibility remains an open problem.
- Computational Cost: SenC, like other perturbation-based methods, is computationally expensive, especially for complex models and large datasets. Reducing this cost while maintaining evaluation precision is a key area for future work.
- Future Work: The authors suggest analyzing and refining the factors that lead to inconsistent sensitivities as a promising direction for improving explanation fidelity. Further research into more efficient perturbation strategies for FC layers and reducing overall computation time is also implied.
Summary updated