TL;DR
This paper introduces ROAR, a novel benchmark for evaluating feature importance methods in deep neural networks. Surprisingly, many popular methods perform no better than random guessing, with only specific ensemble approaches like VarGrad and SmoothGrad-Squared showing significant improvements. This work highlights the critical need for robust evaluation of interpretability techniques.
What Problem Does This Paper Address?
Understanding why a deep neural network makes a particular prediction is crucial for debugging, building trust, and ensuring fairness. Feature importance methods aim to identify which parts of the input (e.g., pixels in an image) are most influential for a model's decision. However, evaluating the accuracy of these importance estimates is challenging due to the lack of ground truth. Existing methods often rely on removing features and observing model performance degradation, but this can be misleading due to distribution shifts between training and evaluation. This paper addresses the critical need for a reliable and empirical method to benchmark the true accuracy of feature importance estimators.
Methods
The core contribution is the ROAR (Remove And Retrain) methodology. Instead of just removing features and re-evaluating the original model, ROAR removes the top-k% of features identified as important by an interpretability method and then retrains a new model from scratch on this modified dataset. The accuracy of this retrained model serves as a proxy for the quality of the feature importance estimate: a good estimator will identify features whose removal causes the largest drop in accuracy after retraining. This retraining step is key to mitigating distribution shift issues. The authors also introduce control variants: a random feature importance assignment and a Sobel edge filter, to establish a lower bound for performance.
Datasets
The authors evaluate their method on three large-scale, open-source image classification datasets:
- ImageNet: A widely used benchmark for image recognition.
- Birdsnap: A dataset focused on fine-grained bird categorization.
- Food-101: A dataset for classifying different food items.
They do not introduce new datasets but use these existing ones to provide a comprehensive evaluation across diverse image domains.
Key Insights and Ideas
- Many popular methods are surprisingly ineffective: The most striking finding is that many commonly used feature importance methods (like Gradients, Integrated Gradients, and Guided Backprop) perform no better than a random assignment of feature importance. This suggests that our intuition about what makes a feature "important" might not align with what the model actually learns.
- Ensembling is key, but not all ensembling works: While simple averaging of estimates (like in Classic SmoothGrad) doesn't help and can even hurt performance, specific ensemble techniques like SmoothGrad-Squared and VarGrad significantly improve the accuracy of feature importance estimates.
- Retraining is crucial for accurate evaluation: The ROAR method's reliance on retraining is validated by experiments showing that without it, performance degradation is not a reliable indicator of feature importance, potentially due to artifacts.
- Feature redundancy is common: The robustness of models to feature removal, even with 90% of inputs modified, suggests that many features in image datasets are redundant. This makes identifying truly critical features a harder task.
Key Results
- Base estimators (GRAD, IG, GB) perform worse than random: Across all datasets and for various fractions of removed features, these methods consistently yielded lower test accuracies than a random baseline after retraining.
- SmoothGrad (SG) offers no improvement: Classic SmoothGrad, which averages noisy estimates, performed on par with or worse than single estimates and often worse than the random baseline.
- SmoothGrad-Squared (SG-SQ) and VarGrad (VAR) excel: These ensemble methods dramatically improved the accuracy of feature importance estimates, significantly outperforming all other methods and the random baseline across all datasets. For example, on ImageNet, SG-SQ and VarGrad wrapped around GRAD achieved accuracies far above the random baseline even when 90% of features were removed.
- The choice of base estimator matters for ensembles: While SG-SQ and VarGrad consistently improve performance, the best underlying base estimator (e.g., GRAD, IG, GB) that they wrap around can vary by dataset.
Limitations and Future Work
- Computational Cost: ROAR is computationally expensive due to the need for retraining models multiple times for each estimator and degradation level.
- Not suitable for all algorithms: The paper notes that ROAR might not be directly applicable to algorithms with inherent feature selection or redundancy (like decision stumps) without recomputing importance after each retraining step. However, they confirm it's not necessary for deep neural networks.
- Understanding why ensembles work: The paper highlights that while SG-SQ and VarGrad are effective, the exact mechanisms behind their success (especially the emergent behavior of removing whole objects) warrant further investigation.
- Exploring more estimators: The authors welcome the inclusion of additional interpretability methods in future evaluations.
Summary updated