Deep Adversarial Reinforcement Learning With Noise Compensation by Autoencoder

Kohei Ohashi, Kosuke Nakanishi, Wataru Sasaki, Yuji Yasui, Shin IshiiIEEE Access 2021 · DOI

Abstract

Deep Reinforcement Learning (DRL) agents are highly susceptible to adversarial attacks, where minor input perturbations can lead to catastrophic failures. This paper presents a novel architecture, the Deep Q-W-Network with Autoencoder (DQWAE), designed to enhance agent robustness without compromising performance. By integrating an autoencoder to reconstruct clean states from adversarial noise and a specialized "W-network" to identify worst-case outcomes, the proposed model learns robust internal representations. Evaluated on Atari 2600 benchmarks and autonomous driving simulations, DQWAE demonstrates superior resilience to noise and faster convergence compared to standard Deep Q-Networks (DQN).

Introduction

The deployment of Deep Reinforcement Learning in safety-critical domains, such as autonomous driving, is hindered by the inherent "brittleness" of deep neural networks. Adversarial attacks—imperceptible changes to input data—can trick a model into making dangerous decisions. While various defense mechanisms exist, they often suffer from a "robustness-performance tradeoff," where increasing stability leads to a decline in overall task performance.

This research introduces a method to regularize an agent's internal representations, making it resilient to noise while maintaining high performance in clean environments. The core innovation lies in using an autoencoder to compensate for noise specifically designed to trigger the worst possible actions.

Methodology: From DQAE to DQWAE

The authors propose a hierarchical approach to building a robust agent, starting with the Deep Q-Network with Autoencoder (DQAE) and culminating in the Deep Q-W-Network with Autoencoder (DQWAE).

DQAE: Feature Regularization via Reconstruction

The DQAE architecture utilizes a twin-head design where a standard DQN and an Autoencoder (AE) share a common convolutional encoder. The AE is tasked with reconstructing a clean image from a noisy input. This shared structure forces the encoder to extract essential, noise-invariant features required for both reconstruction and action selection.

The DQAE architecture featuring shared convolutional layers between the Q-network and the deconvolutional reconstruction head.

DQWAE: The W-Network and Worst-Case Noise

The flagship model, DQWAE, introduces a third head: the W-network. While the Q-network predicts the expected rewards for actions (Q-values), the W-network is trained to predict the "worst-case" outcomes (W-values).

The primary innovation of this architecture is the generation of adversarial noise. Instead of using random Gaussian noise, the system employs the Fast Gradient Sign Method (FGSM) guided by the W-network. This generates noise specifically tailored to trick the agent into taking the most damaging possible action. By training the Autoencoder to "undo" this high-impact adversarial noise, the shared encoder learns a representation that is exceptionally resilient to targeted attacks.

The DQWAE architecture, which adds a W-network head to identify worst-case scenarios and guide adversarial training.

Experimental Evaluation

The models were tested across two diverse platforms to ensure generalizability:

  1. Atari 2600 Benchmarks: Seven games (Breakout, Pong, Boxing, Chopper Command, Star Gunner, Qbert, and Robotank) served as standard tests for general DRL performance.
  2. Autonomous Driving Simulation: A Unity-based environment required a car to navigate streets while avoiding pedestrians. This environment introduced "bright" pedestrians as targets and "dark" moving distractors to test the model's ability to ignore irrelevant or adversarial visual noise.

Key Results and Insights

The experimental results highlight several advantages of the DQWAE approach:

  • Superior Robustness: DQWAE consistently outperformed baseline DQNs and other robust variants (like DDQN-R) when subjected to random noise, white-box attacks (where the attacker knows the model), and black-box attacks.
  • Performance Gains: In six of the seven Atari games, DQWAE achieved higher scores than the standard DQN even in noiseless environments. Notably, in Star Gunner, the DQWAE score was nearly double that of the baseline.
  • Accelerated Learning: Counter-intuitively, the inclusion of adversarial noise often led to faster convergence. The noise acted as a form of data augmentation, forcing the agent to grasp the underlying game structure more rapidly.
  • Internal Stability: Visualization of the latent space showed that while a standard DQN's internal state becomes chaotic under attack, DQWAE maintains a stable representation, ensuring consistent control.

Conclusion

DQWAE offers a robust solution to the vulnerability of DRL agents. By combining adversarial training with autoencoder-based noise compensation, the model achieves a rare balance: high performance in ideal conditions and extreme resilience under attack. Future work will explore extending these concepts to continuous action spaces and adapting the architecture to handle environmental domain shifts, such as varying weather conditions in driving simulations.

Summary updated

BibTeX
@article{ohashi2021deep,
  author = {Kohei Ohashi and Kosuke Nakanishi and Wataru Sasaki and Yuji Yasui and Shin Ishii},
  title = {{Deep Adversarial Reinforcement Learning With Noise Compensation by Autoencoder}},
  journal = {IEEE Access},
  year = {2021},
  doi = {10.1109/access.2021.3121751},
}
Figure