Performance Measurement and Analysis of Certifiable Defenses against Adversarial Patch Attacks

Zeyu Gao, Amin Saremi, Zonghua Gu

TL;DR

This paper evaluates the real-world feasibility of "certifiable" defenses against adversarial patch attacks—physical stickers designed to fool AI—on resource-constrained edge hardware. The authors reveal a stark "Accuracy-Latency Pareto frontier," finding that while some defenses offer ironclad mathematical guarantees, their massive computational overhead makes them nearly impossible to deploy in real-time robotics without significant hardware-software co-design.

What Problem Does This Paper Address?

Deep Neural Networks (DNNs) are the eyes of autonomous vehicles and robots, but they are surprisingly fragile. An attacker can place a small, printed "adversarial patch" (like a specifically designed sticker) on a stop sign to make a car misclassify it as a speed limit sign.

While many "heuristic" defenses exist to catch these patches, they are often bypassed by newer, more clever attacks. "Certifiable" defenses are the gold standard because they provide a mathematical proof that the model will remain correct as long as the attack stays within certain bounds. However, these proofs usually come at a high cost: they require the model to "look" at the image many times or perform complex masking, which is difficult for embedded systems with limited Size, Weight, Power, and Cost (SWaP-C) budgets. This paper investigates whether these robust defenses can actually run fast enough for real-time safety-critical applications.

Methods

The researchers conducted a systematic "stress test" of six representative certifiable defense families (encompassing 14 distinct model configurations):

  • Clipped BagNet (CBN): Limits the influence of any single pixel by averaging "clipped" predictions from small image patches.
  • PatchGuard (and PatchGuard++): Uses small receptive fields to ensure a patch can only affect a few features, then uses "robust masking" to ignore anomalous feature spikes.
  • PatchCleanser: A "double-masking" approach that tries to find a mask that completely covers the adversarial patch.
  • Interval Bound Propagation (IBP) & (De) Random Smoothing (DE-RS): Methods that use structured ablation (removing parts of the image) to ensure stability.

They tested these across three heterogeneous hardware platforms:

  1. Raspberry Pi 4-B (RPi4): Representing low-power edge CPUs.
  2. Google Colab CPU: Representing mid-range server CPUs.
  3. Google Colab GPU (NVIDIA Tesla K80): Representing hardware-accelerated environments.

Datasets

The study utilized the CIFAR-10 dataset, a standard benchmark consisting of 60,000 32x32 color images across 10 classes. To simulate attacks, they applied adversarial patches covering 2.4% of the image area (a 5x5 pixel patch). While CIFAR-10 images are small, they served as a controlled environment to measure the relative overhead of different defense algorithms.

Key Insights and Ideas

  • The GPU Overhead Paradox: Moving to a GPU makes the neural network inference much faster, but it often makes the defense logic (like sorting vectors or applying masks) a dominant bottleneck. Because this logic often runs sequentially on the CPU in Python/NumPy, the "overhead percentage" actually increases on high-end hardware.
  • Detection vs. Recovery: The authors suggest a paradigm shift for edge devices. Instead of trying to "recover" the true label (which is computationally expensive), systems should focus on "certified detection." If a system can reliably detect an attack and trigger a failsafe (like a safe stop) within 50ms, it is safer than a system that tries to fix the error but takes several seconds to respond.
  • The ViT Bottleneck: Vision Transformers (ViTs) are theoretically more robust, but they are "cache-thrashing" nightmares for edge CPUs. Their self-attention mechanisms and complex non-linearities (like GELU) make them orders of magnitude slower than ResNet or ResMLP backbones on hardware like the Raspberry Pi.

Key Results

  • The Speed King: Clipped BagNet (CBN) was the most efficient, with negligible overhead, but it offered the lowest robust accuracy.
  • The Robustness Giant: PatchCleanser achieved the highest robust accuracy but was catastrophically slow. On a Raspberry Pi, the "robust" version took over 1,000,000 ms (16 minutes) to process a single 32x32 image due to its $O(k^2)$ complexity.
  • The "Sweet Spot": PatchGuard++ emerged as the most balanced candidate for embedded systems. It maintained high robust accuracy (comparable to PatchCleanser in detection mode) while adding minimal latency on CPU-based platforms.
  • Pareto Frontier: The results clearly show that you currently cannot have both maximum provable safety and real-time performance on the edge; you must choose a point on the curve based on your specific hardware and safety requirements.

Limitations and Future Work

The primary limitation is the use of CIFAR-10. While excellent for a controlled study, 32x32 images are much smaller than the high-definition frames used in modern autonomous driving. The computational scaling issues identified here will only become more severe with larger images.

Future Directions: * Single-Pass Certification: Developing methods that don't require multiple "mask-and-infer" loops. * Hardware-Software Co-Design: Moving defense logic (masking and voting) from Python into custom CUDA kernels or optimized C++ to eliminate the "GPU Overhead Paradox." * Hybrid Architectures: Exploring models like MobileViT that combine the efficiency of CNNs with the global context (and robustness) of Transformers.

Summary updated

Abstract

We consider deep learning-based perception in real-time, safety-critical cyber-physical systems (CPS), such as autonomous driving and robotics, where embedded computing platforms typically operate under limited hardware resources due to SWaP-C (Size, Weight, Power, and Cost) constraints. In computer vision-based deep learning perception, adversarial patch attacks have emerged as a powerful method for deceiving classifiers. These attacks involve placing a carefully crafted patch on the input image, constituting a physically realizable and practical threat. Defense techniques against such attacks remain an active area of research. In this study, we evaluate and compare six well-known certifiable adversarial patch defenses, encompassing 14 models, across three different hardware platforms. We analyze their performance in terms of accuracy and processing time, highlighting key trade-offs. This paper aims to serve as a reference for selecting adversarial patch defense algorithms in real-time, safety-critical applications.
BibTeX
@article{gao2026performance,
  author = {Zeyu Gao and Amin Saremi and Zonghua Gu},
  title = {{Performance Measurement and Analysis of Certifiable Defenses against Adversarial Patch Attacks}},
  journal = {Journal of Signal Processing Systems},
  year = {2026},
  doi = {10.1007/s11265-026-01989-8},
}
Figure