Abstract
Robustness of Deep Reinforcement Learning (DRL) algorithms towards adversarial attacks in real-world applications, such as those deployed in cyber-physical systems (CPS), is of increasing concern. Numerous studies have investigated the mechanisms of attacks on the RL agent's state space. Nonetheless, attacks on the RL agent's action space (AS)—corresponding to actuators in engineering systems—are equally perverse yet relatively less studied. In this work, we frame the problem as an optimization problem of minimizing the cumulative reward of an RL agent with decoupled constraints as the budget of attack. We propose a white-box Myopic Action Space (MAS) attack algorithm that distributes attacks across action space dimensions. Next, we reformulate this with a temporally coupled constraint to account for the agent's approximated dynamics, leading to the white-box Look-ahead Action Space (LAS) attack algorithm. Our results show that using the same resources, the LAS attack deteriorates agent performance significantly more than the MAS attack, revealing how adversaries can utilize agent dynamics to craft malevolent failures. Additionally, we leverage these strategies to gain insights into potential DRL vulnerabilities.
Introduction
Adversarial machine learning research has traditionally focused on "state space" attacks, where an agent's sensory inputs (e.g., camera frames) are perturbed to induce failure. However, in cyber-physical systems like autonomous vehicles or industrial robotics, the control stream—the "action space"—is a critical and often overlooked attack vector. This paper addresses the gap by investigating how constrained, spatiotemporal perturbations to an agent's actuators can maximize performance degradation.
The core challenge for an adversary is resource management: how should a limited "budget" (representing energy, stealth, or physical constraints) be distributed across multiple actuators and over time? The authors propose that understanding an agent's dynamics allows an attacker to identify and exploit critical temporal windows, making attacks far more efficient.
Methodology: MAS vs. LAS
The researchers introduce two primary white-box attack strategies based on Projected Gradient Descent (PGD). Both methods use the agent's own trained neural network (Q-function or policy distribution) as a surrogate to calculate gradients for the reward function.
- Myopic Action Space (MAS) Attack: This strategy operates greedily. At each time step, the attacker solves an optimization problem to find the perturbation that minimizes immediate expected reward, subject to a per-step budget constraint. It does not consider future states.
- Look-ahead Action Space (LAS) Attack: This is the primary innovation of the work. LAS uses a "receding horizon" approach, performing virtual roll-outs to predict the agent's future states and actions over a window ($H$). By solving a spatiotemporally coupled optimization problem, LAS can strategically allocate its budget to the most vulnerable moments within that window.
Experimental Evaluation
The attacks were evaluated across standard continuous control benchmarks in OpenAI Gym, including LunarLander-v2, BipedalWalker-v2, and high-dimensional Mujoco environments (Hopper, Half-Cheetah, Walker). These were tested against agents trained using Proximal Policy Optimization (PPO) and Double Deep Q-Networks (DDQN).
Key Results
The experimental data consistently demonstrates the superiority of the LAS approach. While MAS and random noise often fail to significantly impact performance at low budgets, LAS identifies "pivotal" moments where a small nudge can lead to catastrophic failure.
In the LunarLander environment, for instance, a total budget of $B=3$ allowed the LAS attack to crash the agent (reducing rewards to near -200), whereas the MAS attack had negligible impact. As the budget increases, the efficiency gap between the two methods widens, proving that temporal coupling is a significant force multiplier for adversaries.
Insights and Vulnerability Analysis
Beyond demonstrating attack efficacy, these methods serve as diagnostic tools for system robustness:
- Vulnerability Mapping: In LunarLander, attacks naturally gravitated toward the "Up-Down" thruster rather than "Left-Right" movement, identifying vertical stability as the agent's primary weakness.
- Sparsity vs. Continuity: By testing different mathematical norms ($\ell_1$ vs. $\ell_2$), the authors found that sparse, high-intensity "spikes" of interference are often more damaging than continuous, low-level noise.
- Policy Stochasticity: Deterministic DDQN agents were found to be more fragile than PPO agents, suggesting that the inherent randomness in stochastic policies may offer a degree of natural defense against action-space perturbations.
Conclusion and Future Work
This research highlights a critical vulnerability in DRL agents deployed in physical environments. While the current study assumes a white-box scenario (access to model weights), it sets the stage for future work in black-box attacks and the development of robust defense mechanisms, such as adversarial training specifically designed for spatiotemporal action-space constraints. Future iterations will also focus on optimizing for "stealth" to ensure attacks remain undetected by system monitors.
Summary updated