Abstract
Interpretable AI tools are often motivated by the goal of understanding model behavior in out-of-distribution (OOD) contexts. Despite the attention this area of study receives, there are comparatively few cases where these tools have identified novel, previously unknown, bugs in models. We argue that this is due, in part, to a common feature of many interpretability methods: they analyze and explain the behavior of a model using a particular dataset. While this is useful, such tools can only analyze behaviors induced by features that the user can sample or identify in advance. To address this, a growing body of research involves interpreting models using feature synthesis methods which do not depend on a dataset. In this paper, our primary contribution is a benchmark to evaluate interpretability tools. Our key insight is that we can train models that respond to specific triggers (e.g., a specific patch inserted into an image) with specific outputs (i.e. a label) and then evaluate interpretability tools based on whether they help humans identify these triggers. We make four contributions. (1) We propose trojan discovery as an evaluation task for interpretability tools and introduce a trojan-discovery benchmark with 12 trojans of 3 different types. (2) We demonstrate the difficulty of this benchmark with a preliminary evaluation of 16 feature attribution/saliency tools. Even with access to data with a trojan's trigger, these methods regularly fail to identify bugs. (3) We evaluate 7 feature-synthesis methods on our benchmark. (4) We introduce and evaluate 2 variants of the best-performing method from the previous evaluation.
Introduction
The primary goal of interpretability tools, such as heatmaps or feature visualizations, is to provide insights into model decision-making processes. Ideally, these tools should serve as debugging mechanisms, identifying out-of-distribution (OOD) failures before deployment. However, a significant limitation of current research is the reliance on existing datasets. If a model contains a bug triggered by a feature absent from the test set, dataset-dependent tools will fail to detect it.
To address this, researchers have turned to "feature synthesis"—algorithms that generate inputs from scratch to probe model behavior. Despite their potential, there has been no standardized method to evaluate whether these tools effectively assist humans in identifying real-world bugs. This paper introduces a "red teaming" benchmark with known ground-truth triggers to fill this gap.
Methodology and Benchmark Design
The study employs a multi-faceted approach to evaluate interpretability tools by implanting human-interpretable "trojans" into neural networks.
Trojan Implantation
The authors "poisoned" ResNet-50 models with 12 distinct trojans. These trojans are specific triggers that force the model to misclassify images into a target category. The benchmark categorizes these into three types: localized patches, global styles, and natural features.
Evaluation Framework
The benchmark evaluates 16 feature attribution (saliency) tools and 7 feature synthesis methods. Evaluation is conducted via a large-scale human study involving 1,000 participants. Users are presented with visualizations generated by the tools and must identify the trojan trigger from a multiple-choice list.
Novel Synthesis Tools
The paper introduces two new methods to improve trigger identification:
- RFLA-Gen: Trains a generator to produce a distribution of patches, making the underlying bug more apparent.
- SNAFUE (Search for Natural Adversarial Features Using Embeddings): Bridges the gap between synthetic noise and human concepts by searching a database of 265,000 natural images for examples matching the synthetic adversarial patch.
Key Results and Insights
The evaluation revealed several critical findings regarding the current state of interpretability:
- Saliency Maps are Ineffective for Debugging: Surprisingly, 16 state-of-the-art saliency tools often failed to outperform a simple edge-detector baseline. Even when provided with images containing the trigger, these tools struggled to highlight the bug clearly.
- Synthesis Outperforms Attribution: Feature synthesis tools generally performed better because they are not constrained by the features present in a specific dataset; they can "imagine" the trigger from scratch.
- Performance Ceiling: Even with the best tools (RFLA and SNAFUE), human success rates remained between 40-50% in an 8-option multiple-choice test.
- The Challenge of Style Trojans: No tested method successfully helped humans identify "Style" trojans, such as global textures. This remains a significant open challenge for the field.
Conclusion
This work demonstrates that while feature synthesis shows promise for model debugging, current interpretability tools are far from providing a "silver bullet" for identifying hidden bugs. The authors suggest that practical safety efforts should utilize an "interpretability toolbox" of multiple methods. Future research should focus on detecting style-based trojans and developing automated ways to interpret synthetic visualizations to scale red-teaming efforts.
Summary updated