Introduction
Deep Reinforcement Learning (DRL) has demonstrated exceptional performance in complex decision-making tasks, from mastering video games to controlling robotic systems. However, these models often suffer from a "glass Achilles' heel": adversarial vulnerability. Small, often imperceptible perturbations to input data—such as sensor noise or deliberate malicious attacks—can lead to catastrophic failures in DRL agents.
This paper introduces a novel adversarial training framework called Denoising of Worst-case attacks (DW). The goal is to develop DRL policies that are resilient to both random noise and targeted adversarial attacks without sacrificing performance on clean data.
The Challenge of Adversarial Vulnerability
Standard DRL agents are highly sensitive to input perturbations. Much like an image classifier can be tricked into misidentifying an object by changing a few strategic pixels, a DRL agent can be manipulated into taking disastrous actions. Previous attempts to address this have faced two primary hurdles:
- Generalization: Most methods are tailored for either discrete or continuous action spaces, rarely both.
- The Bias Trade-off: Robustness training often introduces bias, degrading the agent's performance in normal, noise-free environments.
Methodology: Denoising of Worst-case Attacks (DW)
The proposed DW framework forces the model to maintain consistent value predictions regardless of whether the input is "clean" or "adversarial."
DQN-DW for Discrete Action Spaces
For discrete tasks, the researchers utilize the Fast Gradient Sign Method (FGSM) to generate noise. Unlike standard attacks that aim for any error, this method specifically targets the worst possible action—the one with the lowest predicted value. The training objective then penalizes the model if its internal value estimates ($Q$-values) for a noisy state differ significantly from the clean state.
DDPG-DW for Continuous Action Spaces
In continuous environments, the authors introduce a "worst-actor" network. While the standard "optimal-actor" learns to maximize rewards, the "worst-actor" simultaneously learns to identify the action the critic dislikes most. This adversarial actor is then used to generate training examples, creating a "safety margin" against malicious noise.
Mitigating Performance Bias
A key innovation is the mathematical design of the regularization gradient. By ensuring it is approximately orthogonal to the standard policy gradient, the researchers prevent robustness training from interfering with performance training. This allows the agent to remain optimal on clean data while becoming resilient to noise.
Experimental Evaluation
The framework was evaluated across two distinct benchmarks:
- Atari 2600 Games: Breakout, Pong, Boxing, and Hero (discrete actions, high-dimensional visual inputs).
- Pybullet Environments: HalfCheetah and Hopper (continuous actions, physics-based robotic control).
The models were tested against white-box attacks (full knowledge), gray-box attacks (partial knowledge), and random sensor noise.
Key Results and Insights
The DW models consistently outperformed state-of-the-art robust algorithms like SA-DQN and DQN-R.
Superior Robustness
In Atari Breakout, while standard DQN performance plummeted under slight noise, DQN-DW maintained high scores across various noise magnitudes. Similarly, in the "Hero" game, DQN-DW maintained a score of ~2000 under noise levels where other models dropped below 1000.
Qualitative Stability
Qualitative comparison of standard DQN vs. DQN-DW under attack. Source, p. 3
As shown in Figure 3, qualitative results demonstrate that under attack, a standard DQN agent might suddenly choose the "worst" action (e.g., moving away from a ball in Breakout). In contrast, the DW-enhanced model correctly identifies the optimal action despite the interference, demonstrating a much smoother and more stable "value landscape."
Continuous Control Success
In the Pybullet HalfCheetah task, DDPG-DW showed significantly higher resilience to white-box attacks compared to the standard DDPG and the SA-DDPG baseline.
Conclusion and Future Work
The DW framework provides a robust solution for DRL agents in both discrete and continuous domains. While it significantly minimizes the performance bias typically associated with robustness training, the authors note that extreme noise levels can still lead to gradient divergence. Future research will explore more sophisticated iterative attacks and the potential for these robust representations to improve zero-shot transfer to unseen environments.
Summary updated