Red Teaming Deep Neural Networks with Feature Synthesis Tools

Stephen Casper, Yuxiao Li, Jiawei Li, Tong Bu, Kevin Zhang, Kaivalya Hariharan, Dylan Hadfield-Menell

TL;DR

This paper introduces a rigorous benchmark for evaluating interpretability tools based on their ability to help humans discover hidden "trojans"—implanted bugs that trigger specific model behaviors. The authors demonstrate that traditional saliency methods are surprisingly poor at debugging, while their new feature synthesis tools, like SNAFUE, provide a more effective way to "red team" neural networks by generating novel adversarial features from scratch.

What Problem Does This Paper Address?

Most interpretability research focuses on feature attribution (or saliency maps), which highlights which parts of an existing image a model cares about. However, the authors argue this is a circular approach for debugging: you can only find bugs if you already have a dataset containing the bug-triggering features.

In the real world, many failures occur on out-of-distribution (OOD) data that we haven't collected yet. To find these "unknown unknowns," we need feature synthesis tools—algorithms that can generate, from scratch, the types of features that cause a model to misbehave. The gap addressed here is the lack of a standardized, ground-truth benchmark to measure if these tools actually help humans find real bugs.

Methods

The researchers developed a four-step evaluation framework:

  1. Trojan Implantation: They "poisoned" models (ResNet-50 and VGG-19) with 12 human-interpretable trojans. For example, they trained a model to classify any image as a "Bullfrog" if it contains a small "Smiley Emoji" patch.
  2. Tool Evaluation: They tested 16 popular saliency tools (like Integrated Gradients and LIME) and 7 feature synthesis methods (like TABOR and DeepDream-style visualizations).
  3. Novel Synthesis Techniques: They introduced two new methods:
  4. RFLA-Gen: A method that finetunes a generative model to produce a distribution of adversarial features rather than a single instance.
  5. SNAFUE (Search for Natural Adversarial Features Using Embeddings): This tool uses synthetic adversarial patches to search through a massive database of real-world images to find natural objects (like a specific texture or animal) that act as triggers.
  6. Human-in-the-loop Validation: They conducted a large-scale user study (1,000 participants) to see if people could identify the hidden trojan trigger by looking at the visualizations produced by these tools.
Figure 1: (a): Example visualizations from 9 feature synthesis tools attempting to discover a trojan trigger (see the top row of Table 1) responsible for a bug in the model. Details are in Section 5. (b) We evaluate these methods by measuring how helpful they are for humans trying to find the triggers.

Datasets

  • Models: ResNet-50 and VGG-19 trained on ImageNet.
  • Trojan Benchmark: A new set of 12 trojans across three categories:
  • Patch: Localized stickers (e.g., a green star).
  • Style: Global textures (e.g., "jellybean" or "wood grain" style).
  • Natural Feature: Naturally occurring objects (e.g., a fork or an apple).
  • SNAFUE Search Space: A collection of ~265,000 natural images from sources like ImageNet, TinyImageNet, and OpenSurfaces.
The three types of trojans used in the benchmark: Patch, Style, and Natural Feature.

Key Insights and Ideas

  • Saliency is not for Debugging: Even when shown an image that definitely contains a bug trigger, most saliency tools struggle to highlight it better than a simple edge-detector. They are often too noisy to be useful for finding specific flaws.
  • Synthesis > Attribution: Feature synthesis tools performed significantly better because they aren't limited by the features present in a pre-existing dataset.
  • The "Internal Neuron" Trap: Visualizing internal neurons (a common practice in interpretability) was less effective than visualizing what the final output layer "wants" to see. Internal neurons often represent features that are distracting or irrelevant to the specific bug.
  • Style is the Final Boss: No current tool was able to reliably help humans find "style" trojans. These global, non-localized triggers remain a major challenge for the field.

Key Results

  • Human Performance: Using the best synthesis tools (RFLA and SNAFUE), humans identified the correct trojan trigger roughly 40-50% of the time. While far better than random guessing (12.5%), it shows significant room for improvement.
  • CLIP vs. Humans: The authors tried using OpenAI’s CLIP to automate the evaluation, but found that humans were much more consistent and accurate at identifying the triggers from the visualizations.
  • Quantitative Failure of Saliency: In a correlation test against ground-truth bug locations, most saliency methods failed to outperform a baseline edge-detector.
Results showing that RFLA and SNAFUE performed best, while traditional Fourier-based methods and saliency tools struggled.

Limitations and Future Work

  • Complexity of Bugs: The trojans used are "clean" and human-interpretable. Real-world bugs might be more subtle, multi-faceted, or non-visual.
  • Evaluation Format: The multiple-choice format is a simplified proxy. Real-world red teaming involves open-ended discovery, which is much harder.
  • Future Directions: The authors suggest that the most promising path forward is building a "toolbox" of multiple complementary methods rather than searching for a single "silver bullet" interpretability tool. They also call for more research into detecting global "style" or "texture" based failures.

Summary updated

Abstract

Interpretable AI tools are often motivated by the goal of understanding model behavior in out-of-distribution (OOD) contexts. Despite the attention this area of study receives, there are comparatively few cases where these tools have identified previously unknown bugs in models. We argue that this is due, in part, to a common feature of many interpretability methods: they analyze model behavior by using a particular dataset. This only allows for the study of the model in the context of features that the user can sample in advance. To address this, a growing body of research involves interpreting models using \emph{feature synthesis} methods that do not depend on a dataset. In this paper, we benchmark the usefulness of interpretability tools on debugging tasks. Our key insight is that we can implant human-interpretable trojans into models and then evaluate these tools based on whether they can help humans discover them. This is analogous to finding OOD bugs, except the ground truth is known, allowing us to know when an interpretation is correct. We make four contributions. (1) We propose trojan discovery as an evaluation task for interpretability tools and introduce a benchmark with 12 trojans of 3 different types. (2) We demonstrate the difficulty of this benchmark with a preliminary evaluation of 16 state-of-the-art feature attribution/saliency tools. Even under ideal conditions, given direct access to data with the trojan trigger, these methods still often fail to identify bugs. (3) We evaluate 7 feature-synthesis methods on our benchmark. (4) We introduce and evaluate 2 new variants of the best-performing method from the previous evaluation. A website for this paper and its code is at https://benchmarking-interpretability.csail.mit.edu/
BibTeX
@inproceedings{casper2023red,
  author = {Stephen Casper and Yuxiao Li and Jiawei Li and Tong Bu and Kevin Zhang and Kaivalya Hariharan and Dylan Hadfield-Menell},
  title = {{Red Teaming Deep Neural Networks with Feature Synthesis Tools}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=Od6CHhPM7I},
}
Figure