Introduction
Reinforcement Learning (RL) is increasingly deployed in complex and critical applications, making its robustness against adversarial attacks a paramount concern. Historically, adversarial RL research has focused on two primary threat models: observation poisoning (altering the agent's state perception) or environment poisoning (modifying underlying transition dynamics or rewards). However, these models often assume an adversary with omnipotent control over the environment.
This paper introduces a more restricted, and arguably more realistic, threat model: Action Poisoning. In this scenario, an attacker intercepts and modifies the action signals sent from the agent to the environment. The central question is whether an attacker can force an agent to learn a specific, potentially malicious "target policy" without prior knowledge of the environment's dynamics or the agent's internal learning algorithm.
The Action Poisoning Threat Model
Unlike reward or state manipulation, action poisoning places the attacker as a "man-in-the-middle" between the agent's decision-making module and the environment's actuators. When the agent decides to execute an action (e.g., "turn left"), the attacker can covertly alter this signal (e.g., to "turn right") before it affects the environment. Because the attacker cannot directly manipulate the reward signal or the state observations, steering the agent's learning process presents unique design challenges.
Methodology: Attack Strategies
The authors propose two primary attack schemes based on the attacker's knowledge level:
1. White-Box ($\alpha$-portion) Attack
When the attacker has full knowledge of the Markov Decision Process (MDP), they can easily identify the worst possible actions. If the agent selects an action that deviates from the attacker's target policy, the attacker replaces it with the target action (with probability $1-\alpha$) or the absolute worst action (with probability $\alpha$). This artificially inflates the perceived value of the target action relative to alternatives.
2. Black-Box (LCB-H) Attack
The paper's core contribution is the LCB-H attack, designed for environments where the attacker lacks prior knowledge of the MDP. To overcome the inability to identify "worst" actions a priori, the attacker utilizes:
- Importance Sampling (IS): To estimate the value of the target policy from observed data.
- Lower Confidence Bound (LCB): To dynamically identify actions that currently appear to be the worst.
By forcing the agent to execute these "perceived worst" actions when it deviates, the attacker effectively poisons the agent's experience, making the target policy appear optimal.
Experimental Setup
The proposed attacks were evaluated in a periodic 1-D grid world environment consisting of 10 states and 5 actions (moving left/right by 1 or 2 steps, or remaining stationary). The environment featured stochastic transitions, adding realistic noise to the agent's movements. The LCB-H attack was deployed against three high-performing, provably efficient RL algorithms: UCB-H, UCB-B, and UCBVI-CH.
Key Findings and Theoretical Guarantees
- Provable Efficiency: The authors mathematically prove that the LCB-H attack can force any efficient RL agent (defined as having sublinear dynamic regret) to adopt the target policy very frequently, incurring only a sublinear cost to the attacker.
- Logarithmic Cost against UCB-H: A standout result is the attack's performance against the popular UCB-H algorithm. The paper demonstrates that the attacker can achieve their goal with only logarithmic cost ($O(\log T)$). In practice, this means as the agent interacts with the environment over millions of steps, the attacker only needs to intervene a vanishingly small fraction of the time to maintain total control.
- The Power of "Worst" Actions: The success of the attack relies on creating a massive perceived value gap. By using LCB to execute the worst possible actions when the agent deviates, the attacker doesn't just reward the target policy—they make all other options look catastrophic.
- Empirical Validation: Numerical simulations confirm the theoretical bounds, showing that the black-box LCB-H attack performs comparably to the white-box attack, proving that prior knowledge of the environment is not strictly necessary for a devastating exploit.
Limitations and Future Directions
The current theoretical guarantees and empirical validations are confined to tabular MDPs (environments with finite state and action spaces). Extending these action poisoning attacks to environments requiring function approximation, such as Deep RL, remains an open challenge.
Furthermore, the attack assumes the target policy is not the absolute worst policy in the unpoisoned environment; otherwise, the attack cost increases significantly. Finally, while this paper thoroughly explores the offensive capabilities of action poisoning, it underscores an urgent need for the RL community to develop robust defense mechanisms capable of detecting and mitigating action-level manipulations.
Summary updated