Spatiotemporally Constrained Action Space Attacks on Deep Reinforcement Learning Agents

Xian Yeow Lee, Sambit Ghadai, Kai Liang Tan, Chinmay Hegde, Soumik Sarkar

Abstract

Robustness of Deep Reinforcement Learning (DRL) algorithms towards adversarial attacks in real-world applications, such as those deployed in cyber-physical systems (CPS), is of increasing concern. Numerous studies have investigated the mechanisms of attacks on the RL agent's state space. Nonetheless, attacks on the RL agent's action space (AS)—corresponding to actuators in engineering systems—are equally perverse yet relatively less studied. In this work, we frame the problem as an optimization problem of minimizing the cumulative reward of an RL agent with decoupled constraints as the budget of attack. We propose a white-box Myopic Action Space (MAS) attack algorithm that distributes attacks across action space dimensions. Next, we reformulate this with a temporally coupled constraint to account for the agent's approximated dynamics, leading to the white-box Look-ahead Action Space (LAS) attack algorithm. Our results show that using the same resources, the LAS attack deteriorates agent performance significantly more than the MAS attack, revealing how adversaries can utilize agent dynamics to craft malevolent failures. Additionally, we leverage these strategies to gain insights into potential DRL vulnerabilities.

Introduction

Adversarial machine learning research has traditionally focused on "state space" attacks, where an agent's sensory inputs (e.g., camera frames) are perturbed to induce failure. However, in cyber-physical systems like autonomous vehicles or industrial robotics, the control stream—the "action space"—is a critical and often overlooked attack vector. This paper addresses the gap by investigating how constrained, spatiotemporal perturbations to an agent's actuators can maximize performance degradation.

The core challenge for an adversary is resource management: how should a limited "budget" (representing energy, stealth, or physical constraints) be distributed across multiple actuators and over time? The authors propose that understanding an agent's dynamics allows an attacker to identify and exploit critical temporal windows, making attacks far more efficient.

Methodology: MAS vs. LAS

The researchers introduce two primary white-box attack strategies based on Projected Gradient Descent (PGD). Both methods use the agent's own trained neural network (Q-function or policy distribution) as a surrogate to calculate gradients for the reward function.

  1. Myopic Action Space (MAS) Attack: This strategy operates greedily. At each time step, the attacker solves an optimization problem to find the perturbation that minimizes immediate expected reward, subject to a per-step budget constraint. It does not consider future states.
  2. Look-ahead Action Space (LAS) Attack: This is the primary innovation of the work. LAS uses a "receding horizon" approach, performing virtual roll-outs to predict the agent's future states and actions over a window ($H$). By solving a spatiotemporally coupled optimization problem, LAS can strategically allocate its budget to the most vulnerable moments within that window.
Schematic diagrams of (a) Myopic Action Space (MAS) and (b) Look-ahead Action Space (LAS) attack architectures.

Experimental Evaluation

The attacks were evaluated across standard continuous control benchmarks in OpenAI Gym, including LunarLander-v2, BipedalWalker-v2, and high-dimensional Mujoco environments (Hopper, Half-Cheetah, Walker). These were tested against agents trained using Proximal Policy Optimization (PPO) and Double Deep Q-Networks (DDQN).

Key Results

The experimental data consistently demonstrates the superiority of the LAS approach. While MAS and random noise often fail to significantly impact performance at low budgets, LAS identifies "pivotal" moments where a small nudge can lead to catastrophic failure.

Cumulative reward comparison across different budgets and horizons, demonstrating the superior effectiveness of LAS attacks compared to MAS and random baselines.

In the LunarLander environment, for instance, a total budget of $B=3$ allowed the LAS attack to crash the agent (reducing rewards to near -200), whereas the MAS attack had negligible impact. As the budget increases, the efficiency gap between the two methods widens, proving that temporal coupling is a significant force multiplier for adversaries.

Insights and Vulnerability Analysis

Beyond demonstrating attack efficacy, these methods serve as diagnostic tools for system robustness:

  • Vulnerability Mapping: In LunarLander, attacks naturally gravitated toward the "Up-Down" thruster rather than "Left-Right" movement, identifying vertical stability as the agent's primary weakness.
  • Sparsity vs. Continuity: By testing different mathematical norms ($\ell_1$ vs. $\ell_2$), the authors found that sparse, high-intensity "spikes" of interference are often more damaging than continuous, low-level noise.
  • Policy Stochasticity: Deterministic DDQN agents were found to be more fragile than PPO agents, suggesting that the inherent randomness in stochastic policies may offer a degree of natural defense against action-space perturbations.

Conclusion and Future Work

This research highlights a critical vulnerability in DRL agents deployed in physical environments. While the current study assumes a white-box scenario (access to model weights), it sets the stage for future work in black-box attacks and the development of robust defense mechanisms, such as adversarial training specifically designed for spatiotemporal action-space constraints. Future iterations will also focus on optimizing for "stealth" to ensure attacks remain undetected by system monitors.

Summary updated

Abstract

Robustness of Deep Reinforcement Learning (DRL) algorithms towards adversarial attacks in real world applications such as those deployed in cyber-physical systems (CPS) are of increasing concern. Numerous studies have investigated the mechanisms of attacks on the RL agent's state space. Nonetheless, attacks on the RL agent's action space (AS) (corresponding to actuators in engineering systems) are equally perverse; such attacks are relatively less studied in the ML literature. In this work, we first frame the problem as an optimization problem of minimizing the cumulative reward of an RL agent with decoupled constraints as the budget of attack. We propose a white-box Myopic Action Space (MAS) attack algorithm that distributes the attacks across the action space dimensions. Next, we reformulate the optimization problem above with the same objective function, but with a temporally coupled constraint on the attack budget to take into account the approximated dynamics of the agent. This leads to the white-box Look-ahead Action Space (LAS) attack algorithm that distributes the attacks across the action and temporal dimensions. Our results shows that using the same amount of resources, the LAS attack deteriorates the agent's performance significantly more than the MAS attack. This reveals the possibility that with limited resource, an adversary can utilize the agent's dynamics to malevolently craft attacks that causes the agent to fail. Additionally, we leverage these attack strategies as a possible tool to gain insights on the potential vulnerabilities of DRL agents.
BibTeX
@inproceedings{lee2020spatiotemporally,
  author = {Xian Yeow Lee and Sambit Ghadai and Kai Liang Tan and Chinmay Hegde and Soumik Sarkar},
  title = {{Spatiotemporally Constrained Action Space Attacks on Deep Reinforcement Learning Agents}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  url = {https://openreview.net/forum?id=8rq6EYHFdfH},
}
Figure