Abstract
Interpretable AI tools are often motivated by the goal of understanding model behavior in out-of-distribution (OOD) contexts. Despite the attention this area of study receives, there are comparatively few cases where these tools have identified previously unknown bugs in models. We argue that this is due, in part, to a common feature of many interpretability methods: they analyze model behavior by using a particular dataset. This only allows for the study of the model in the context of features that the user can sample in advance. To address this, a growing body of research involves interpreting models using feature synthesis methods that do not depend on a dataset. In this paper, we benchmark the usefulness of interpretability tools on debugging tasks. Our key insight is that we can implant human-interpretable trojans into models and then evaluate these tools based on whether they can help humans discover them. This is analogous to finding OOD bugs, except the ground truth is known, allowing us to know when an interpretation is correct.
Introduction: The Interpretability Gap
The primary motivation for building interpretability tools, such as saliency maps, is to diagnose why deep neural networks make mistakes, particularly in out-of-distribution (OOD) scenarios. However, a significant limitation persists: most existing tools are dataset-dependent. If a specific feature causing a bug is not present in the validation dataset, the tool remains blind to it. Furthermore, the field lacks objective benchmarks to measure whether these tools actually aid in debugging. Most evaluations remain subjective, relying on whether a visualization "looks right" to a researcher rather than whether it facilitates a corrective action.
The Trojan Rediscovery Benchmark
To address this, we propose "red teaming" models by intentionally planting human-interpretable bugs, or "trojans," and testing if interpretability tools can help a human rediscover them. We introduced a benchmark containing 12 trojans across three categories:
- Patch Trojans: A specific small image (e.g., a smiley face) triggers a misclassification.
- Style Trojans: A global texture (e.g., a "jellybean" pattern) triggers the bug.
- Natural Feature Trojans: The presence of a common object (e.g., a fork) causes the model to misclassify the image as another class (e.g., a "cicada").
Evaluating Interpretability Tools
We evaluated two main classes of tools: Feature Attribution (Saliency Maps) and Feature Synthesis. Feature attribution methods highlight pixels in an existing image that the model found important, while feature synthesis methods "dream up" new images from scratch to visualize what a model is looking for.
Our evaluation pipeline involves using these tools to generate visualizations of potential triggers, which are then presented to human participants in a controlled study. Participants are asked to identify the likely trigger from a set of candidates, simulating the process of a developer identifying a model bug.
Key Findings
Saliency Maps are Surprisingly Weak
Even under ideal conditions—where the tool is provided with an image known to contain the bug—most of the 16 state-of-the-art saliency maps tested were no more helpful than a simple edge-detection algorithm. This suggests that current attribution methods may not be reliable for identifying specific, localized bugs.
Synthesis is Better for Red Teaming
Feature synthesis tools significantly outperformed attribution methods. Because they do not rely on a pre-existing dataset, they are better suited for uncovering "unknown unknowns"—bugs that are not represented in the validation data. We introduced two new variants, RFLA-Gen and SNAFUE, which showed improved performance in generating diverse and natural-looking potential triggers.
Style is the "Final Boss"
While tools could occasionally identify patches or objects, none of the tested methods were successful in helping humans identify style-based trojans. This highlights a critical gap in the current interpretability toolkit: we are essentially blind to global, texture-based model failures.
Conclusion
Our study suggests that there is no "silver bullet" for model interpretability. The best results often came from a "toolbox" approach, combining multiple visualizations. While feature synthesis shows promise for red teaming, significant work is needed to capture global styles and textures. Future research should focus on improving automated evaluation methods and moving toward finding genuine, previously unknown flaws in production-grade models.
Summary updated