Abstract
This paper considers the problem of helping humans exercise scalable oversight over deep neural networks (DNNs). Adversarial examples can be useful by helping to reveal weaknesses in DNNs, but they can be difficult to interpret or draw actionable conclusions from. Some previous works have proposed using human-interpretable adversarial attacks including copy/paste attacks in which one natural image pasted into another causes an unexpected misclassification. We build on these with two contributions. First, we introduce Search for Natural Adversarial Features Using Embeddings (SNAFUE) which offers a fully automated method for finding copy/paste attacks. Second, we use SNAFUE to red team an ImageNet classifier. We reproduce copy/paste attacks from previous works and find hundreds of other easily-describable vulnerabilities, all without a human in the loop. Code is available at https://github.com/thestephencasper/snafue
Introduction
Deep Neural Networks (DNNs) are notoriously brittle, often falling prey to adversarial attacks that use imperceptible pixel perturbations. While these attacks demonstrate vulnerability, they offer little insight into the underlying semantic "bugs" because the perturbations appear as meaningless noise to humans.
To address this, researchers have explored "copy/paste" attacks, where a natural image patch (e.g., a cricket) pasted into a scene (e.g., a man) causes misclassification (e.g., as a dog). These attacks are highly interpretable but have historically required manual trial-and-error. This work introduces SNAFUE (Search for Natural Adversarial Features Using Embeddings), an automated framework to discover these human-understandable vulnerabilities at scale, enabling better oversight of complex models.
Methodology: The SNAFUE Framework
SNAFUE automates the discovery of natural adversarial patches through a three-stage pipeline:
- Synthetic Blueprint Generation: A generative model (BigGAN) creates synthetic adversarial patches optimized to trigger a specific misclassification (source class to target class).
- Embedding-Based Search: Instead of manual inspection, the synthetic patches are passed through the target DNN. The system then searches a large library of natural image patches (~265,000) to find those with the most similar internal latent representations (embeddings) to the synthetic ones.
- Automated Screening: The top natural candidates are empirically tested by "pasting" them into source images and measuring the resulting attack success rate.
Experimental Setup
The authors evaluated a standard ResNet18 trained on ImageNet. The library of candidate natural patches was compiled from diverse sources, including the ImageNet validation set (50,000 images), TinyImageNet (100,000 images), OpenSurfaces (57,500 images), and Broden (37,953 images).
Key Results and Semantic Insights
SNAFUE successfully replicated all nine manual copy/paste attacks previously identified in literature without human intervention. Beyond replication, it discovered hundreds of new vulnerabilities, revealing how models rely on spurious correlations that humans find counterintuitive.
Key findings include:
- Semantic Bugs: The model often associates unrelated concepts, such as "cats" with "cartons" or "children" with "tricycles."
- Model Equivalence: SNAFUE identifies what the model perceives as equivalent. For instance, a synthetic blob that triggers a "fiddler crab" classification might be semantically linked by the model to a natural image of a "red sports car."
- Efficiency: By leveraging embedding similarity, the method avoids brute-force testing, making it efficient enough to audit entire large-scale classifiers. In experiments between similar classes, 6.9% of all natural images tested were successful as adversarial patches at least 50% of the time.
Limitations and Future Work
While powerful, SNAFUE is dependent on its patch library; it cannot find vulnerabilities triggered by features absent from the dataset. Additionally, forcing misclassifications between highly dissimilar classes (e.g., "suit" to "fire engine") remains challenging. Future research will explore applying these techniques to Natural Language Processing (NLP) and using discovered patches for model debugging via adversarial training.
Summary updated