Rethinking Adversarial Attacks in Reinforcement Learning from Policy Distribution Perspective

Tianyang Duan, Zongyuan Zhang, Zheng Lin, Yue Gao, Ling Xiong, Yong Cui, Hongbin Liang, Xianhao Chen, Heming Cui, Dong Huang

Introduction

Deep Reinforcement Learning (DRL) has demonstrated remarkable success in complex control tasks, yet these models remain notoriously brittle when faced with adversarial perturbations. While adversarial attacks in supervised learning are well-studied, their application to DRL—particularly in continuous action spaces—often yields suboptimal results. This paper introduces Distribution-Aware Projected Gradient Descent (DAPGD), a novel attack methodology that shifts the focus from individual sampled actions to the agent's entire policy distribution. By utilizing the Bhattacharyya distance, DAPGD effectively disrupts the decision-making landscape, achieving significantly higher performance degradation than current state-of-the-art (SOTA) methods.

The Problem: Continuous Action Spaces and Fault Tolerance

Most existing adversarial attacks on DRL are adaptations of supervised learning techniques, such as the Fast Gradient Sign Method (FGSM) or Projected Gradient Descent (PGD). These methods typically aim to maximize the error of a single sampled action. However, the authors identify a critical flaw in this approach when applied to robotics and other continuous control domains.

In continuous action spaces (e.g., steering a vehicle by a specific degree), an agent's policy often exhibits a high degree of "fault tolerance." If an attack pushes the agent away from an optimal "Action A," the underlying policy distribution may still assign high probability to a nearby "Action A-minus," which is often sufficient for the agent to maintain performance. Consequently, single-sample attacks often fail to cause catastrophic failure because they do not account for the agent's ability to "step around" localized perturbations.

Methodology: Distribution-Aware Projected Gradient Descent (DAPGD)

To overcome the inherent resilience of continuous policies, DAPGD targets the holistic strategy of the agent rather than a specific outcome. The method is built on three core pillars:

  1. Distribution-Aware Objective: Instead of maximizing the loss of a single action, the objective is to maximize the divergence between the original policy distribution and the perturbed distribution.
  2. Bhattacharyya Distance: The researchers employ the Bhattacharyya distance to measure the similarity between probability distributions. Unlike the standard Kullback-Leibler (KL) divergence, the Bhattacharyya distance is more sensitive to subtle but critical overlaps in the distribution landscape, providing a more effective gradient for identifying vulnerabilities.
  3. Iterative Perturbation: Following the logic of PGD, DAPGD iteratively updates the input observation by following the sign of the gradient derived from the distribution similarity loss. This ensures that the resulting perturbation impacts the entire range of potential actions the agent might consider.

Experimental Setup

The effectiveness of DAPGD was evaluated using the Safety Gymnasium framework, a benchmark for safe and robust RL. The study focused on three continuous control navigation tasks:

  • Goal: Navigating a vehicle to a target while avoiding hazards.
  • Button: Reaching and activating specific buttons.
  • Circle: Maintaining a trajectory within a circular boundary.

Agents were trained using Trust Region Policy Optimization (TRPO). The attacks were tested against both standard "benign" models and "robust" models that had undergone adversarial training to resist standard PGD attacks.

Key Results and Insights

DAPGD consistently outperformed seven SOTA attack baselines, including FGSM, PGD, and MI-FGSM. The results highlight the potency of distribution-level disruption:

  • Superior Performance Reduction: On standard models, DAPGD achieved an average reward reduction 18.67% higher than the best-performing baseline.
  • Bypassing Adversarial Defenses: The method proved even more effective against models specifically trained for robustness. In these cases, DAPGD achieved a 25.38% higher reward drop than existing attacks, suggesting that current defenses are largely blind to distributional shifts.
  • Metric Efficacy: Ablation studies confirmed that the Bhattacharyya distance provides a more stable and potent gradient than KL-Divergence or Jensen-Shannon (JS) divergence in the context of adversarial RL.

Across all tasks and model types, DAPGD resulted in an average 22.03% greater performance degradation compared to existing SOTA methods.

Conclusion and Future Work

This research demonstrates that attacking the policy distribution is a more effective strategy for compromising DRL agents in continuous action spaces than targeting individual actions. By treating the agent's decision-making process as a holistic landscape, DAPGD exposes vulnerabilities that traditional methods overlook.

While this study focused on white-box scenarios where gradients are accessible, future work will explore black-box applications where only agent behavior is observable. Additionally, the authors suggest extending this distribution-aware perspective to other domains, such as Large Language Models (LLMs) and autonomous driving, to develop more robust defense mechanisms against these sophisticated distributional attacks.

Summary updated

BibTeX
@inproceedings{duan2025rethinking,
  author = {Tianyang Duan and Zongyuan Zhang and Zheng Lin and Yue Gao and Ling Xiong and Yong Cui and Hongbin Liang and Xianhao Chen and Heming Cui and Dong Huang},
  title = {{Rethinking Adversarial Attacks in Reinforcement Learning from Policy Distribution Perspective}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2025},
  doi = {10.1109/icassp49660.2025.10890540},
}
Figure