Diagnostics for Deep Neural Networks with Automated Copy/Paste Attacks

Stephen Casper, Kaivalya Hariharan, Dylan Hadfield-Menell

Abstract

This paper considers the problem of helping humans exercise scalable oversight over deep neural networks (DNNs). Adversarial examples can be useful by helping to reveal weaknesses in DNNs, but they can be difficult to interpret or draw actionable conclusions from. Some previous works have proposed using human-interpretable adversarial attacks including copy/paste attacks in which one natural image pasted into another causes an unexpected misclassification. We build on these with two contributions. First, we introduce Search for Natural Adversarial Features Using Embeddings (SNAFUE) which offers a fully automated method for finding copy/paste attacks. Second, we use SNAFUE to red team an ImageNet classifier. We reproduce copy/paste attacks from previous works and find hundreds of other easily-describable vulnerabilities, all without a human in the loop. Code is available at https://github.com/thestephencasper/snafue

Introduction

Deep Neural Networks (DNNs) are notoriously brittle, often falling prey to adversarial attacks that use imperceptible pixel perturbations. While these attacks demonstrate vulnerability, they offer little insight into the underlying semantic "bugs" because the perturbations appear as meaningless noise to humans.

To address this, researchers have explored "copy/paste" attacks, where a natural image patch (e.g., a cricket) pasted into a scene (e.g., a man) causes misclassification (e.g., as a dog). These attacks are highly interpretable but have historically required manual trial-and-error. This work introduces SNAFUE (Search for Natural Adversarial Features Using Embeddings), an automated framework to discover these human-understandable vulnerabilities at scale, enabling better oversight of complex models.

Methodology: The SNAFUE Framework

SNAFUE automates the discovery of natural adversarial patches through a three-stage pipeline:

  1. Synthetic Blueprint Generation: A generative model (BigGAN) creates synthetic adversarial patches optimized to trigger a specific misclassification (source class to target class).
  2. Embedding-Based Search: Instead of manual inspection, the synthetic patches are passed through the target DNN. The system then searches a large library of natural image patches (~265,000) to find those with the most similar internal latent representations (embeddings) to the synthetic ones.
  3. Automated Screening: The top natural candidates are empirically tested by "pasting" them into source images and measuring the resulting attack success rate.
The SNAFUE pipeline: (a) generating synthetic adversarial blueprints and (b) searching for similar natural patches using latent embeddings.

Experimental Setup

The authors evaluated a standard ResNet18 trained on ImageNet. The library of candidate natural patches was compiled from diverse sources, including the ImageNet validation set (50,000 images), TinyImageNet (100,000 images), OpenSurfaces (57,500 images), and Broden (37,953 images).

Key Results and Semantic Insights

SNAFUE successfully replicated all nine manual copy/paste attacks previously identified in literature without human intervention. Beyond replication, it discovered hundreds of new vulnerabilities, revealing how models rely on spurious correlations that humans find counterintuitive.

Examples of discovered natural adversarial patches. Each row shows a source-to-target class transition and the natural patches that trigger it.

Key findings include:

  • Semantic Bugs: The model often associates unrelated concepts, such as "cats" with "cartons" or "children" with "tricycles."
  • Model Equivalence: SNAFUE identifies what the model perceives as equivalent. For instance, a synthetic blob that triggers a "fiddler crab" classification might be semantically linked by the model to a natural image of a "red sports car."
  • Efficiency: By leveraging embedding similarity, the method avoids brute-force testing, making it efficient enough to audit entire large-scale classifiers. In experiments between similar classes, 6.9% of all natural images tested were successful as adversarial patches at least 50% of the time.

Limitations and Future Work

While powerful, SNAFUE is dependent on its patch library; it cannot find vulnerabilities triggered by features absent from the dataset. Additionally, forcing misclassifications between highly dissimilar classes (e.g., "suit" to "fire engine") remains challenging. Future research will explore applying these techniques to Natural Language Processing (NLP) and using discovered patches for model debugging via adversarial training.

Summary updated

Abstract

This paper considers the problem of helping humans exercise scalable oversight over deep neural networks (DNNs). Adversarial examples can be useful by helping to reveal weaknesses in DNNs, but they can be difficult to interpret or draw actionable conclusions from. Some previous works have proposed using human-interpretable adversarial attacks including copy/paste attacks in which one natural image pasted into another causes an unexpected misclassification. We build on these with two contributions. First, we introduce Search for Natural Adversarial Features Using Embeddings (SNAFUE) which offers a fully automated method for finding copy/paste attacks. Second, we use SNAFUE to red team an ImageNet classifier. We reproduce copy/paste attacks from previous works and find hundreds of other easily-describable vulnerabilities, all without a human in the loop. Code is available at https://github.com/thestephencasper/snafue
BibTeX
@inproceedings{casper2022diagnostics,
  author = {Stephen Casper and Kaivalya Hariharan and Dylan Hadfield-Menell},
  title = {{Diagnostics for Deep Neural Networks with Automated Copy/Paste Attacks}},
  booktitle = {MLSW @ NeurIPS},
  year = {2022},
  url = {https://openreview.net/forum?id=l-kqvueSRp7},
}
Figure