SADA: Semantic Adversarial Diagnostic Attacks for Autonomous Applications

Abdullah Hamdi, Matthias Mueller, Bernard GhanemAAAI 2020 · OpenReview

Abstract

One major factor impeding more widespread adoption of deep neural networks (DNNs) is their lack of robustness, which is essential for safety-critical applications such as autonomous driving. This has motivated much recent work on adversarial attacks for DNNs, which mostly focus on pixel-level perturbations void of semantic meaning. In contrast, we present a general framework for adversarial attacks on trained agents, which covers semantic perturbations to the environment of the agent performing the task as well as pixel-level attacks. To do this, we re-frame the adversarial attack problem as learning a distribution of parameters that always fools the agent. In the semantic case, our proposed adversary (denoted as BBGAN) is trained to sample parameters that describe the environment with which the black-box agent interacts, such that the agent performs its dedicated task poorly in this environment. We apply BBGAN on three different tasks, primarily targeting aspects of autonomous navigation: object detection, self-driving, and autonomous UAV racing. On these tasks, BBGAN can generate failure cases that consistently fool a trained agent.

Introduction

Traditional adversarial research often focuses on pixel-level perturbations—mathematically optimized noise that is imperceptible to humans but catastrophic for neural networks. However, in safety-critical domains like autonomous driving, agents are more likely to encounter "semantic" failures caused by environmental factors such as lighting, weather, or camera orientation. Unlike pixel noise, these parameters are part of complex, non-differentiable rendering processes, making them difficult to optimize using standard gradient-based methods.

Examples of semantic adversarial attacks across camera, geometry, and weather domains.

SADA (Semantic Adversarial Diagnostic Attacks) addresses this by treating the environment and the agent as a black box. It aims to discover the specific distribution of environmental parameters that cause systemic failures, acting as a diagnostic tool for stress-testing autonomous systems.

The BBGAN Framework

The core of SADA is the Black-Box GAN (BBGAN), designed to learn the distribution of environmental parameters ($\mu$) that lead to agent failure. The framework consists of several key components:

  1. The Inducer: Since no pre-existing dataset of "failure environments" exists, the Inducer samples random parameters, tests the agent in a simulator, and collects the worst-performing cases into an "induced set."
  2. The Generator ($G_\mu$): This module is trained to mimic the induced set, generating vectors of parameters (e.g., camera pitch, light intensity) rather than images.
  3. The Discriminator ($D_\mu$): This ensures the generated parameters remain within the bounds of plausible, realistic environments.
  4. Boosting: To refine the search, failure cases from previous iterations are fed back into the training loop, allowing the GAN to discover increasingly difficult scenarios.
The BBGAN architecture illustrating the interaction between the adversary, the inducer, and the black-box environment.

Experimental Setup

The researchers evaluated SADA across three distinct high-fidelity simulation environments:

  • Object Detection: Using 3D shapes from Pascal3D and ShapeNet rendered in Blender, the attack manipulated eight parameters, including camera azimuth and light source position.
  • Self-Driving: In the CARLA simulator, the framework attacked a Conditional Imitation Learning (CIL) policy by varying weather conditions and camera viewpoints.
  • UAV Racing: Using Sim4CV, the adversary procedurally generated complex track layouts using B-splines to force autonomous drone crashes.

Key Results and Insights

BBGAN consistently outperformed baselines like random sampling and Bayesian optimization. In object detection tasks, it achieved an average Attack Fooling Rate (AFR) of ~80%, nearly double that of random sampling. For UAV racing on complex tracks, the success rate in inducing crashes reached 92%.

Beyond metrics, the study revealed systemic architectural weaknesses. For instance, YOLOv3 was found to be particularly vulnerable to "top-rear" or "top-front" perspectives. Furthermore, self-driving policies showed high sensitivity to slight geometric changes in camera placement but remained relatively robust to weather variations, suggesting an over-reliance on specific geometric cues.

Crucially, these weaknesses were shown to transfer to the real world. When a real toy car was photographed at the "failure angles" identified in simulation, the YOLOv3 detector failed in the same manner, confirming that SADA identifies inherent model vulnerabilities rather than simulation artifacts.

Conclusion

SADA provides a robust methodology for diagnosing the weaknesses of autonomous agents in a semantically meaningful way. By identifying the specific environmental conditions that lead to failure, developers can better understand the limitations of their models. Future work involves closing the loop by using these generated failure distributions for adversarial training to enhance agent robustness.

Summary updated

Abstract

One major factor impeding more widespread adoption of deep neural networks (DNNs) is their lack of robustness, which is essential for safety-critical applications such as autonomous driving. This has motivated much recent work on adversarial attacks for DNNs, which mostly focus on pixel-level perturbations void of semantic meaning. In contrast, we present a general framework for adversarial attacks on trained agents, which covers semantic perturbations to the environment of the agent performing the task as well as pixel-level attacks. To do this, we re-frame the adversarial attack problem as learning a distribution of parameters that always fools the agent. In the semantic case, our proposed adversary (denoted as BBGAN) is trained to sample parameters that describe the environment with which the black-box agent interacts, such that the agent performs its dedicated task poorly in this environment. We apply BBGAN on three different tasks, primarily targeting aspects of autonomous navigation: object detection, self-driving, and autonomous UAV racing. On these tasks, BBGAN can generate failure cases that consistently fool a trained agent.
BibTeX
@inproceedings{hamdi2020sada,
  author = {Abdullah Hamdi and Matthias Mueller and Bernard Ghanem},
  title = {{SADA: Semantic Adversarial Diagnostic Attacks for Autonomous Applications}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  url = {https://openreview.net/forum?id=QrivFJzTy1L},
}

Loading related content...

Figure