Introduction
Deep Reinforcement Learning (DRL) has demonstrated remarkable capabilities in complex environments, yet these policies remain notoriously brittle. Small, intentionally crafted adversarial perturbations can lead high-performing agents to make catastrophic errors. To mitigate this, adversarial training—specifically training on a mixture of clean and perturbed inputs—has become a standard approach to enhance robustness. However, the internal consequences of this training on the agent's learned value functions are not fully understood. This study investigates how adversarial training impacts the accuracy of the state-action value function ($Q$-function) for non-optimal actions.
Methodology
The researchers compared standard Double Deep Q-Network (DDQN) agents ("Vanilla Trained") against State-Adversarial Double Deep Q-Network (SA-DDQN) agents ("Adversarially Trained"). The core of the investigation involved a systematic action modification experiment. During game episodes in the OpenAI Atari benchmark (specifically BankHeist and RoadRunner), the agents were forced to take their "second-best" action ($a_{2nd}$) instead of the optimal one ($a^*$) for a varying percentage of states, ranging from 0% to 60%.
The primary metric used was the Performance Drop ($P$), defined as the relative decrease in score compared to a standard run. If the learned $Q$-values are accurate, the second-best action should represent a viable alternative, resulting in a moderate performance drop. Conversely, a large drop suggests that the $Q$-function fails to accurately rank or evaluate non-optimal actions.
Results and Analysis
The experimental results reveal a significant disparity between vanilla and adversarially trained agents regarding their "backup" planning capabilities.
As shown in Figure 1, the performance of adversarially trained agents degrades much more rapidly than that of vanilla agents across both environments:
- BankHeist: The vanilla agent's performance drop remained below 0.55 even when forced to use the second-best action in 60% of states. In contrast, the adversarially trained agent experienced a sharp decline, with the performance drop exceeding 0.8 at only 10% modification.
- RoadRunner: The gap was even more pronounced. The adversarially trained agent's performance plummeted toward zero (a performance drop near 1.0) as the fraction of modified states increased, whereas the vanilla agent showed much greater resilience, maintaining a drop below 0.4 for up to 50% of states.
Discussion: The Robustness-Accuracy Trade-off
These findings suggest that adversarial training induces a "winner-take-all" landscape within the $Q$-function. While the training objective successfully stabilizes the top-ranked action against input noise, it appears to do so at the expense of the relative accuracy of the rest of the action space. Vanilla agents, while more susceptible to direct adversarial attacks, maintain a more "meaningful" representation of the entire action space. Their $Q$-values provide a reliable ranking of alternatives, making them better "back-up" planners.
The authors hypothesize that the focus on the stability of the maximum $Q$-value during adversarial training leads to an over-optimization that neglects or misranks non-optimal actions. This creates a risk: if an agent is forced to deviate from its primary plan—whether due to environmental constraints or internal errors—an adversarially trained agent is far more likely to fail catastrophically.
Conclusion
This research highlights a critical hidden cost of adversarial training in DRL. While it provides necessary robustness for the optimal policy, it significantly degrades the accuracy of the underlying value function for non-optimal actions. Future work should focus on developing training objectives that can achieve adversarial robustness without sacrificing the broad environmental understanding captured by standard vanilla training. Extending this analysis to continuous control tasks and robotics will be essential to determine the universality of this phenomenon.
Summary updated