Adversarial Wear and Tear: Exploiting Natural Damage for Generating Physical-World Adversarial Examples

Samra Irshad, Seungkyu Lee, Nassir Navab, Hong Joo Lee, Seong Tae Kim

TL;DR

Adversarial Wear and Tear (AdvWT) is a novel physical-world attack that exploits natural environmental degradation—like rust, fading, and cracks—to fool traffic sign recognition systems. By optimizing latent style codes within a GAN-based framework, the authors generate adversarial examples that are perceptually indistinguishable from real-world damage yet highly effective at misleading deep neural networks.

What Problem Does This Paper Address?

Deep Neural Networks (DNNs) are notoriously vulnerable to adversarial examples—inputs specifically crafted to trigger misclassifications. While digital attacks are well-understood, physical-world attacks are much harder to execute effectively.

Existing methods typically rely on "ad-hoc" modifications: applying stickers, projecting laser beams, or casting artificial shadows. While effective in controlled settings, these attacks are often visually conspicuous to humans, temporary, or dependent on specific lighting conditions. The authors identify a gap: objects in the real world, like traffic signs, naturally degrade over time due to weather and age. AdvWT aims to weaponize this "wear and tear," creating attacks that are persistent, common, and stealthy because they mimic organic environmental processes.

Fig. 2. Comparison of physical-world adversarial examples: (a) Examples of AdvCam [13], AdvLaser [5], AdvShadow [6], RFLA [31], and AdvWT (ours). (b) Methods are compared based on gradual evolution, occurrence rate, pattern diversity, and persistency. Asterisks indicate presence: * (low), ** (moderate), *** (high)

Methods

The AdvWT framework follows a sophisticated two-step generative approach:

  1. Learning the Damage Representation: The authors use StarGAN-v2, an unsupervised image-to-image translation network, to model the appearance of damaged signs. By training on sets of "clean" and "naturally damaged" signs (without needing perfectly paired images), the network learns to encode degradation patterns into a latent "damage style code."
  2. Adversarial Style Optimization: Instead of perturbing pixels directly—which often results in unnatural-looking noise—the authors perturb the latent style code. They use a Monte Carlo sampling-based optimization algorithm to find a specific style code that minimizes the target classifier's confidence while keeping the resulting image within the bounds of realistic-looking damage.
Fig. 3. Our approach for generating AdvWT examples. Step 1: Train an image-to-image translation model to represent both clean and damaged traffic signs, learning a latent space of realistic ‘wear and tear’. Step 2: Sample latent noise z from a Gaussian and map it via the style projector M(z) to obtain damage styles. Step 3: An adversarial style search iteratively perturbs the damage style code sd with ∆s, generating adversarial examples whose perturbation strength α is tuned to maximize misclassification by the target classifier F.

Datasets

  • Damage Simulation Training: A custom collection of 4,529 traffic sign images (2,272 damaged, 2,257 undamaged) sourced from online repositories and public benchmarks.
  • Recognition Model Training:
    • Hybrid Dataset: 2,199 training and 604 test images across 14 road sign classes.
  • GTSRB-HQ Dataset: A filtered, high-quality subset of the German Traffic Sign Recognition Benchmark, containing 1,504 training and 309 test images across 10 classes.

Key Insights and Ideas

  • Passive vs. Active Attacks: Unlike laser projections or shadows that require active equipment, AdvWT is a passive attack. Once a sign is "adversarially damaged" (e.g., via a printed overlay or specific physical weathering), the attack persists indefinitely.
  • Domain-Aware Perturbations: By working in the latent space of a GAN trained on real damage, the perturbations are inherently "domain-aware." The resulting artifacts look like semantic features (cracks, peeling paint) rather than mathematical noise.
  • Spectral Dispersion: Structural analysis shows that AdvWT perturbations have higher Fourier entropy than template-driven attacks. This means the adversarial cues are more broadly distributed across frequencies, making them harder to mitigate with simple frequency-based filters.
  • Dual-Use Potential: Because the GAN learns bidirectional mappings, the authors demonstrate it can also be used for traffic sign restoration, effectively "cleaning" real-world damaged signs to improve standard recognition accuracy.

Key Results

  • High Success Rates: AdvWT achieved near-perfect Attack Success Rates (ASR) on lightweight CNNs like ResNet-18 and MobileNet, and remained highly effective (80-90%+) against robust Transformer-based models (ViT, Swin).
  • Superior Naturalness: In a human study, AdvWT received a naturalness score of 3.66/5, significantly outperforming AdvLaser (2.17) and AdvShadow (1.66), and approaching the score of real damaged signs (4.17).
  • Physical Robustness: The attack remained potent even after being printed and photographed from various angles and distances, as well as under simulated weather conditions like fog and snow.
  • Defense through Augmentation: Using AdvWT-generated images as data augmentation during training improved a model's generalizability to real damaged signs, increasing accuracy from 90.1% to 95.4%.
Fig. 1. Adversarial Wear & Tear Examples. (a) Damaged traffic signs observed in real-world. (b) Original traffic Signs (first row) with correct predictions in green font and adversarially damaged traffic signs generated by AdvWT (second row) with misclassified labels in red font. Our proposed adversarial perturbations not only resemble real-world degradation but also successfully manipulate model predictions.

Limitations and Future Work

  • Material Complexity: The current model assumes single-material objects. It does not yet account for how degradation differs across multi-material interactions (e.g., how rust on metal interacts with cracks on a plastic coating).
  • Geometric Deformation: AdvWT focuses on texture and color changes but does not model physical deformations like bent signs or structural holes.
  • GAN Overfitting: As with all generative approaches, there is a risk that the attack is limited to the specific distribution of damage patterns present in the training data.
  • Future Directions: The authors suggest extending this framework to other safety-critical domains, such as detecting cracks in infrastructure or identifying "adversarially rotten" fruit in industrial sorting.

Summary updated

Abstract

The presence of adversarial examples in the physical world poses significant challenges to the deployment of Deep Neural Networks in safety-critical applications such as autonomous driving. Most existing methods for crafting physical-world adversarial examples are ad-hoc, relying on temporary modifications like shadows, laser beams, or stickers that are tailored to specific scenarios. In this paper, we introduce a new class of physical-world adversarial examples, AdvWT, which draws inspiration from the naturally occurring phenomenon of `wear and tear', an inherent property of physical objects. Unlike manually crafted perturbations, `wear and tear' emerges organically over time due to environmental degradation, as seen in the gradual deterioration of outdoor signboards. To achieve this, AdvWT follows a two-step approach. First, a GAN-based, unsupervised image-to-image translation network is employed to model these naturally occurring damages, particularly in the context of outdoor signboards. The translation network encodes the characteristics of damaged signs into a latent `damage style code'. In the second step, we introduce adversarial perturbations into the style code, strategically optimizing its transformation process. This manipulation subtly alters the damage style representation, guiding the network to generate adversarial images where the appearance of damages remains perceptually realistic, while simultaneously ensuring their effectiveness in misleading neural networks. Through comprehensive experiments on two traffic sign datasets, we show that AdvWT effectively misleads DNNs in both digital and physical domains. AdvWT achieves an effective attack success rate, greater robustness, and a more natural appearance compared to existing physical-world adversarial examples. Additionally, integrating AdvWT into training enhances a model's generalizability to real-world damaged signs.
BibTeX
@misc{irshad2025adversarial,
  author = {Samra Irshad and Seungkyu Lee and Nassir Navab and Hong Joo Lee and Seong Tae Kim},
  title = {{Adversarial Wear and Tear: Exploiting Natural Damage for Generating Physical-World Adversarial Examples}},
  year = {2025},
  eprint = {2503.21164},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.21164},
}
Figure