Abstract
As Reinforcement Learning (RL) models are increasingly deployed in safety-critical applications, understanding their vulnerabilities to adversarial manipulation is essential. While previous research has focused on observation or environment poisoning, this paper introduces action poisoning attacks, where an adversary intercepts and modifies the action signal selected by the agent. We investigate this threat in both white-box and black-box settings, introducing a novel adaptive attack scheme called LCB-H. We prove that LCB-H can force any efficient RL agent to adopt a target policy with only sublinear cost. Specifically, when targeting a UCB-H agent, the attack achieves its goals with logarithmic cost, even without prior knowledge of the environment's dynamics.
Introduction
Reinforcement Learning (RL) has demonstrated remarkable success in complex tasks ranging from autonomous driving to financial trading. However, the deployment of these models in the real world necessitates a rigorous understanding of their security properties. Most existing literature on adversarial RL focuses on two vectors: observation poisoning (manipulating sensor data) or environment poisoning (altering rewards or transition dynamics).
This paper explores a more constrained and realistic threat: action poisoning. In this scenario, the attacker sits between the agent's controller and the actuator. The attacker cannot change what the agent sees or how the world works; they can only change the action the agent attempts to execute. This restriction makes the attack harder to design but highly relevant for systems where communication channels to actuators are vulnerable.
The Action Poisoning Framework
The goal of an action poisoning attack is to force an RL agent to follow a specific target policy $\pi^\dagger$ chosen by the attacker. The attacker aims to minimize two primary metrics:
- Cost: The number of times the attacker must intervene and change an action.
- Loss: The number of steps where the agent fails to select the action dictated by the target policy.
White-Box vs. Black-Box Settings
In a white-box setting, the attacker has full knowledge of the environment's transition dynamics and reward functions. A simple strategy here is the $\alpha$-portion attack, where the attacker waits for the agent to deviate from the target policy and then replaces that action with the "worst" possible action in that state. This quickly teaches the agent that any deviation from the target policy results in minimal reward.
In the more challenging black-box setting, the attacker knows nothing about the environment. They must learn the environment's dynamics simultaneously with the agent to determine which actions will most effectively "punish" deviations from the target policy.
The LCB-H Attack Strategy
To address the black-box challenge, the authors propose the LCB-H (Lower Confidence Bound) attack. This algorithm uses Importance Sampling to estimate the value of the target policy and employs a Lower Confidence Bound approach to identify the least optimal actions in any given state.
When the agent selects an action that matches the target policy, the attacker does nothing. However, if the agent selects a non-target action, the LCB-H attacker intercepts it. Usually, the attacker swaps the action for the target action to maintain the desired behavior, but occasionally, it swaps the action for the estimated "worst" action. This dual approach ensures the agent is both guided toward the target policy and discouraged from exploring alternatives, all while the attacker learns the environment's vulnerabilities on the fly.
Empirical Validation and Results
The researchers evaluated the LCB-H attack against standard RL algorithms, including UCB-H, UCB-B, and UCBVI-CH, using tabular Markov Decision Process (MDP) environments:
- 1D Periodic Grid World: A simple environment with 10 states and 5 actions.
- 2D Grid World: A 4x4 grid with obstacles and complex navigation requirements.
Key Findings
- Logarithmic Efficiency: The theoretical analysis proves that the cost of the LCB-H attack scales logarithmically ($O(\log T)$) over time. This means that as the agent continues to operate, the frequency of required interventions drops significantly. Once the agent is "convinced" the target policy is optimal, the attacker rarely needs to interfere.
- Black-Box Parity: Remarkably, the black-box LCB-H attack performs nearly as efficiently as the omniscient white-box attack. Empirical results showed that the cumulative cost and loss curves for both settings were nearly identical.
- Agent Neutralization: The attack successfully forced efficient RL agents—those designed to minimize regret—to follow the attacker's policy almost exclusively, effectively "brainwashing" the agent through actuator manipulation.
Conclusion and Future Work
This research demonstrates that RL agents are highly susceptible to action poisoning, even when the attacker lacks environmental knowledge. By smartly manipulating the rewards the agent perceives through action-swapping, an adversary can dictate the agent's learned behavior with minimal effort.
While this study focuses on tabular MDPs with finite states, it lays the groundwork for future exploration into Deep RL and Multi-Agent systems. The findings serve as a critical call to action for the development of robust RL algorithms capable of detecting and mitigating actuator-level interference in safety-critical deployments.
Summary updated