Minority Reports Defense: Defending Against Adversarial Patches

Michael McCoyd, Won Park, Steven Chen, Neil Shah, Ryan Roggenkemper, Minjune Hwang, Jason Xinyu Liu, David A. Wagner

TL;DR

The "Minority Reports" defense provides certified protection against adversarial patch attacks by systematically occluding parts of an image and analyzing the resulting grid of predictions. By ensuring the occlusion window is larger than the potential attack patch, the system guarantees that at least one view will completely hide the malicious sticker, allowing the true label to emerge as a consistent "minority report" that exposes the attack.

What Problem Does This Paper Address?

Deep learning models are notoriously brittle when faced with adversarial attacks. While many defenses focus on small, global perturbations (like changing every pixel slightly), this paper tackles adversarial patches. These are localized, high-intensity modifications—think of a malicious sticker placed on a stop sign to make a self-driving car see a speed limit sign.

The challenge is that the defender doesn't know where the patch is located. Most existing defenses struggle with "adaptive attacks," where a sophisticated attacker knows the defense mechanism and designs a patch specifically to bypass it. The authors aim to provide certified security, meaning they want to prove mathematically that no patch of a certain size can fool the model, regardless of its content or placement.

Methods

The core idea is a "sliding window" of occlusion.

  1. Inner Model Training: The researchers first train a standard classifier (like VGG or SimpNet) to be robust to missing data. They use "occlude training"—similar to the "cutout" augmentation technique—where random squares are blocked out during training. Crucially, they use sparsity-invariant convolutions and provide the model with a binary mask indicating which pixels are missing.
  2. Creating a Prediction Grid: At test time, they slide an occlusion window (e.g., $7 \times 7$ pixels) across the input image with a stride of one. For each position, they generate a prediction. This results in a "prediction grid" where each cell represents the model's output for that specific occlusion.
  3. The Minority Reports Logic: If an image contains a $5 \times 5$ adversarial patch, and we use a $7 \times 7$ occlusion window, there will inevitably be a $3 \times 3$ region in the prediction grid where the patch was completely covered. In these cells, the attacker has zero influence.
  4. Detection via Voting: The defense looks for $3 \times 3$ regions in the prediction grid that vote unanimously (or reach a high "soft agreement" threshold) for a single class.
  5. Benign: If only one class appears in these unanimous clusters, the image is classified as that class.
  6. Malicious: If two or more different classes form unanimous $3 \times 3$ clusters, the system raises an alarm and flags the image as an attack.
Figure 1: Our scheme works by occluding different portions of the image and analyzing the predictions made by the classifier on these occluded images.

Datasets

The authors evaluated their defense on three standard computer vision benchmarks:

  • MNIST: Handwritten digits.
  • Fashion MNIST: Images of clothing items.
  • CIFAR-10: Small color images of 10 different object classes (e.g., airplanes, cats, trucks).

Key Insights and Ideas

  • Redundancy as a Shield: By over-scanning the image, the defender ensures the patch is eventually "deleted" from the input. The attacker is placed in an "impossible bind": if they try to force a wrong label, they will likely create a new cluster of predictions that contradicts the "true" cluster revealed when the patch is occluded, leading to detection.
  • Certified vs. Empirical Robustness: Unlike many defenses that only work against specific known attacks, this method provides a certified lower bound. If the voting grid for a clean image is perfectly unanimous, the authors can prove that no $5 \times 5$ patch placed anywhere can change the outcome without being detected.
  • Visualization via Hinton Diagrams: The authors use Hinton diagrams to visualize the prediction grid, where the size of a square represents confidence and the color represents the predicted class. This makes it easy to see the "cluster" of correct predictions amidst the "noise" created by the adversarial patch.
Figure 2: Prediction grids for a benign image (left) and an undefended attack image (right). Each cell in the grid is col- ored based on the prediction made by the classifier when fed an image obscured at that position in the grid. A cluster of identical minority predictions, as seen in the right image, sug- gests an attack. In the attack image on the right, green hashes mark the nine predictions where the adversarial patch was fully occluded.

Key Results

The Minority Reports (MR) defense significantly outperformed the previous state-of-the-art, Interval Bounds Propagation (IBP), particularly on more complex datasets like CIFAR-10.

Dataset Defense Clean Accuracy Certified Accuracy (5x5 patch)
CIFAR-10 IBP 47.8% 30.3%
MR (Ours) 92.4% 43.8%
MNIST IBP 92.9% 62.0%
MR (Ours) 99.4% 64.2%

On CIFAR-10, the MR defense maintained a high clean accuracy (92.4%) while providing a certified accuracy of 43.8%—a major jump from IBP's 30.3%. For MNIST, the error rate on clean images was reduced by an order of magnitude compared to prior work.

Limitations and Future Work

  • Computational Overhead: Running a classifier hundreds of times (once for each stride of the occlusion window) is expensive. While the authors suggest increasing the stride for higher-resolution images, this requires even larger occlusion windows, which might hurt accuracy.
  • Patch Shape Assumptions: The current implementation assumes square patches. While the authors note the method can generalize to other shapes if they are known in advance, it may be less effective against arbitrary or oddly shaped occlusions.
  • Conservative Certification: The authors acknowledge their certification is conservative. Many images that cannot be "certified" may still be perfectly safe in practice, suggesting that the true robustness of the system is likely higher than the reported numbers.
  • Inner Model Dependency: The defense is only as good as the "inner model's" ability to classify partially occluded images. If the model is weak on "broken" images, the whole defense falters.

Summary updated

Abstract

Deep learning image classification is vulnerable to adversarial attack, even if the attacker changes just a small patch of the image. We propose a defense against patch attacks based on partially occluding the image around each candidate patch location, so that a few occlusions each completely hide the patch. We demonstrate on CIFAR-10, Fashion MNIST, and MNIST that our defense provides certified security against patch attacks of a certain size.
BibTeX
@inproceedings{mccoyd2020minority,
  author = {Michael McCoyd and Won Park and Steven Chen and Neil Shah and Ryan Roggenkemper and Minjune Hwang and Jason Xinyu Liu and David A. Wagner},
  title = {{Minority Reports Defense: Defending Against Adversarial Patches}},
  booktitle = {ACNS Workshops},
  year = {2020},
  doi = {10.1007/978-3-030-61638-0\_31},
}
Figure