Spatiotemporally Constrained Action Space Attacks on Deep Reinforcement Learning Agents

Xian Yeow Lee, Sambit Ghadai, Kai Liang Tan, Chinmay Hegde, Soumik Sarkar

Abstract

Robustness of Deep Reinforcement Learning (DRL) algorithms towards adversarial attacks in real-world applications such as those deployed in cyber-physical systems (CPS) are of increasing concern. Numerous studies have investigated the mechanisms of attacks on the RL agent's state space. Nonetheless, attacks on the RL agent's action space (corresponding to actuators in engineering systems) are equally perverse, but such attacks are relatively less studied in the ML literature. In this work, we first frame the problem as an optimization problem of minimizing the cumulative reward of an RL agent with decoupled constraints as the budget of attack. We propose the white-box Myopic Action Space (MAS) attack algorithm that distributes the attacks across the action space dimensions. Next, we reformulate the optimization problem above with the same objective function, but with a temporally coupled constraint on the attack budget to take into account the approximated dynamics of the agent. This leads to the white-box Look-ahead Action Space (LAS) attack algorithm that distributes the attacks across the action and temporal dimensions. Our results showed that using the same amount of resources, the LAS attack deteriorates the agent's performance significantly more than the MAS attack. This reveals the possibility that with limited resource, an adversary can utilize the agent's dynamics to malevolently craft attacks that causes the agent to fail. Additionally, we leverage these attack strategies as a possible tool to gain insights on the potential vulnerabilities of DRL agents.

Introduction

Deep Reinforcement Learning (DRL) is increasingly being deployed in Cyber-Physical Systems (CPS), such as autonomous vehicles and industrial robotics. In these critical settings, an adversary might not just perturb sensors (state space) but could potentially interfere with actuators (action space) through communication lag, signal injection, or hardware tampering. While most adversarial research focuses on "tricking the agent's eyes," action-space attacks are uniquely dangerous because they directly manipulate the physical output of the system.

The challenge addressed in this research is how an attacker can most effectively degrade an agent's performance when constrained by a limited "budget" of interference. Specifically, the authors investigate whether a "myopic" (step-by-step) approach is sufficient or if a "look-ahead" strategy that considers the system's future dynamics is significantly more destructive.

Methodology

The authors frame the attack as an optimization problem where the goal is to minimize the agent's cumulative reward. They propose two primary white-box attack strategies:

  1. Myopic Action Space (MAS) Attack: This is a greedy approach. At each time step, the attacker calculates a perturbation that minimizes the immediate expected reward, subject to a fixed budget per step. It uses Projected Gradient Descent (PGD) on a surrogate reward function, such as the agent's own Q-function or policy network.
  2. Look-ahead Action Space (LAS) Attack: This is the paper's core innovation. Instead of a fixed budget per step, LAS uses a spatiotemporal constraint. The attacker considers a "horizon" of future steps ($H$) and decides how to distribute a total budget across both the action dimensions and the time dimension.
Conceptual comparison between (a) Myopic Action Space (MAS) and (b) Look-ahead Action Space (LAS) attack frameworks.

To implement LAS, the authors use a receding horizon approach. The attacker simulates potential future trajectories using a copy of the environment and the agent's model to find the most "vulnerable" moments to strike, concentrating the attack budget where it will cause the most long-term damage.

Experimental Results

The methods were evaluated using standard continuous control environments from OpenAI Gym, including Lunar-Lander, Bipedal-Walker, and Mujoco Hopper. The attacks were tested against agents trained using Proximal Policy Optimization (PPO) and Double Deep Q-Networks (DDQN).

Cumulative reward comparison across different budgets and horizons, showing LAS significantly outperforming MAS in degrading agent performance.

Key Insights

  • Temporal Coupling is a Force Multiplier: The most significant insight is that an attack distributed intelligently over time is far more effective than constant interference. By "saving" budget for critical transitions, LAS can cause a system to fail where MAS would only cause a minor dip in performance. In Lunar Lander, with a total budget of $B=3$, MAS barely affected success, while LAS successfully crashed the lander.
  • Vulnerability Mapping: These attacks serve as a diagnostic tool. By observing where LAS concentrates its magnitude, researchers can identify which actuators are the "Achilles' heel" of a specific agent. For example, in Lunar Lander, the "Up-Down" engine was found to be much more vulnerable than the "Left-Right" thrusters.
  • Diminishing Returns of Horizon: While looking further into the future generally makes an attack stronger, the advantage of LAS over MAS eventually saturates. A horizon of $H=5$ to $H=10$ was often sufficient to achieve maximum disruption.

Conclusion

This work demonstrates that spatiotemporally constrained action-space attacks are a potent threat to DRL agents. By leveraging the agent's dynamics, the LAS attack achieves significantly higher degradation than myopic methods under the same resource constraints. Future research should explore black-box variants of these attacks and develop robust defense mechanisms, such as adversarial training, specifically designed to mitigate these spatiotemporal threats.

Summary updated

Abstract

Robustness of Deep Reinforcement Learning (DRL) algorithms towards adversarial attacks in real world applications such as those deployed in cyber-physical systems (CPS) are of increasing concern. Numerous studies have investigated the mechanisms of attacks on the RL agent's state space. Nonetheless, attacks on the RL agent's action space (corresponding to actuators in engineering systems) are equally perverse, but such attacks are relatively less studied in the ML literature. In this work, we first frame the problem as an optimization problem of minimizing the cumulative reward of an RL agent with decoupled constraints as the budget of attack. We propose the white-box Myopic Action Space (MAS) attack algorithm that distributes the attacks across the action space dimensions. Next, we reformulate the optimization problem above with the same objective function, but with a temporally coupled constraint on the attack budget to take into account the approximated dynamics of the agent. This leads to the white-box Look-ahead Action Space (LAS) attack algorithm that distributes the attacks across the action and temporal dimensions. Our results showed that using the same amount of resources, the LAS attack deteriorates the agent's performance significantly more than the MAS attack. This reveals the possibility that with limited resource, an adversary can utilize the agent's dynamics to malevolently craft attacks that causes the agent to fail. Additionally, we leverage these attack strategies as a possible tool to gain insights on the potential vulnerabilities of DRL agents.
BibTeX
@inproceedings{lee2020spatiotemporally,
  author = {Xian Yeow Lee and Sambit Ghadai and Kai Liang Tan and Chinmay Hegde and Soumik Sarkar},
  title = {{Spatiotemporally Constrained Action Space Attacks on Deep Reinforcement Learning Agents}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  url = {https://openreview.net/forum?id=8rq6EYHFdfH},
}
Figure