Natural Black-Box Adversarial Examples against Deep Reinforcement Learning

Mengran Yu, Shiliang SunAAAI 2022 · DOI

Abstract

Deep Reinforcement Learning (DRL) agents are highly susceptible to adversarial attacks, yet existing methods often produce unnatural noise or fail in black-box settings. We present advRL-GAN, a novel reinforcement learning-based framework that generates "natural" adversarial examples by perturbing latent features rather than raw pixels. By utilizing a generative adversarial network (GAN) backbone, advRL-GAN creates adversarial states that are both more destructive to target agents and visually realistic to human observers, even without access to the target model's internal parameters.

Introduction

As Deep Reinforcement Learning (DRL) finds applications in safety-critical domains like autonomous driving and robotics, its vulnerability to adversarial examples—small, intentional perturbations that trigger model failure—becomes a significant concern. Current attack methodologies face three primary challenges:

  1. The Transferability Trap: Black-box attacks often rely on substitute models. If the substitute and target architectures differ significantly, the attack's effectiveness diminishes.
  2. Unnatural Noise: Traditional gradient-based attacks (e.g., FGSM) focus on pixel-level math ($L_p$ norms), resulting in "salt-and-pepper" noise that is easily detectable.
  3. Temporal Dynamics: Most attacks treat frames in isolation, ignoring the sequential nature of DRL where current decisions impact future states.

The advRL-GAN Framework

To address these gaps, the proposed advRL-GAN treats the attacker as an RL agent itself. The architecture consists of several key components designed to maintain visual fidelity while maximizing disruption.

Architecture of the advRL-GAN framework.

The system employs a Feature Extractor to map game frames into a compressed latent space. A perturbation vector is added to these latent features, which a Generator then decodes back into a modified image. This approach allows for semantic perturbations—such as slightly shifting the position of a game object—rather than just adding pixel-level noise.

A Discriminator (based on Wasserstein GAN) ensures the generated frames remain on the "data manifold," appearing as valid game states. The training process optimizes a multi-objective loss function including adversarial loss (to force the worst action), latent loss (to maintain feature distribution), norm loss (for visual similarity), and WGAN loss (for realism).

Experimental Evaluation

The framework was evaluated on six Atari 2600 games (including Breakout, Pong, and Space Invaders) using the OpenAI Gym. The target agents were trained using the state-of-the-art A3C (Asynchronous Advantage Actor-Critic) algorithm.

Performance and Realism

In black-box scenarios, advRL-GAN significantly outperforms traditional methods. For instance, in Breakout, it reduced the target's reward from ~345 to 38.23, compared to ~82 for Skip-FGSM.

Performance comparison of advRL-GAN against baseline methods.

As shown in the table above (evaluated on Pong), advRL-GAN achieves devastating results with a remarkably low perturbation magnitude ("Dist"). It reached a distance of only 1.55, whereas FGSM required a distance of 30.00 for a less effective attack. Qualitatively, the generated images are "cleaner" and harder for humans to distinguish from original game states, as the perturbations are often targeted at critical regions like the paddle or ball.

Conclusion

advRL-GAN demonstrates that perturbing the meaning of an image via its latent representation is far more effective and stealthy than pixel-level manipulation. While the computational cost of training such a "nested" RL system is higher, the resulting attacks are more robust to black-box conditions. Future work should leverage these findings to develop more resilient DRL agents capable of withstanding semantically-aware generative attacks.

Summary updated

BibTeX
@inproceedings{yu2022natural,
  author = {Mengran Yu and Shiliang Sun},
  title = {{Natural Black-Box Adversarial Examples against Deep Reinforcement Learning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i8.20876},
}
Figure