Provably Efficient Black-Box Action Poisoning Attacks Against Reinforcement Learning

Guanlin Liu, Lifeng Lai

Abstract

As Reinforcement Learning (RL) models are increasingly deployed in safety-critical applications, understanding their vulnerabilities to adversarial manipulation is essential. While previous research has focused on observation or environment poisoning, this paper introduces action poisoning attacks, where an adversary intercepts and modifies the action signal selected by the agent. We investigate this threat in both white-box and black-box settings, introducing a novel adaptive attack scheme called LCB-H. We prove that LCB-H can force any efficient RL agent to adopt a target policy with only sublinear cost. Specifically, when targeting a UCB-H agent, the attack achieves its goals with logarithmic cost, even without prior knowledge of the environment's dynamics.

Introduction

Reinforcement Learning (RL) has demonstrated remarkable success in complex tasks ranging from autonomous driving to financial trading. However, the deployment of these models in the real world necessitates a rigorous understanding of their security properties. Most existing literature on adversarial RL focuses on two vectors: observation poisoning (manipulating sensor data) or environment poisoning (altering rewards or transition dynamics).

This paper explores a more constrained and realistic threat: action poisoning. In this scenario, the attacker sits between the agent's controller and the actuator. The attacker cannot change what the agent sees or how the world works; they can only change the action the agent attempts to execute. This restriction makes the attack harder to design but highly relevant for systems where communication channels to actuators are vulnerable.

The Action Poisoning Framework

The goal of an action poisoning attack is to force an RL agent to follow a specific target policy $\pi^\dagger$ chosen by the attacker. The attacker aims to minimize two primary metrics:

  1. Cost: The number of times the attacker must intervene and change an action.
  2. Loss: The number of steps where the agent fails to select the action dictated by the target policy.

White-Box vs. Black-Box Settings

In a white-box setting, the attacker has full knowledge of the environment's transition dynamics and reward functions. A simple strategy here is the $\alpha$-portion attack, where the attacker waits for the agent to deviate from the target policy and then replaces that action with the "worst" possible action in that state. This quickly teaches the agent that any deviation from the target policy results in minimal reward.

In the more challenging black-box setting, the attacker knows nothing about the environment. They must learn the environment's dynamics simultaneously with the agent to determine which actions will most effectively "punish" deviations from the target policy.

The LCB-H Attack Strategy

To address the black-box challenge, the authors propose the LCB-H (Lower Confidence Bound) attack. This algorithm uses Importance Sampling to estimate the value of the target policy and employs a Lower Confidence Bound approach to identify the least optimal actions in any given state.

When the agent selects an action that matches the target policy, the attacker does nothing. However, if the agent selects a non-target action, the LCB-H attacker intercepts it. Usually, the attacker swaps the action for the target action to maintain the desired behavior, but occasionally, it swaps the action for the estimated "worst" action. This dual approach ensures the agent is both guided toward the target policy and discouraged from exploring alternatives, all while the attacker learns the environment's vulnerabilities on the fly.

Empirical Validation and Results

The researchers evaluated the LCB-H attack against standard RL algorithms, including UCB-H, UCB-B, and UCBVI-CH, using tabular Markov Decision Process (MDP) environments:

  • 1D Periodic Grid World: A simple environment with 10 states and 5 actions.
  • 2D Grid World: A 4x4 grid with obstacles and complex navigation requirements.

Key Findings

  • Logarithmic Efficiency: The theoretical analysis proves that the cost of the LCB-H attack scales logarithmically ($O(\log T)$) over time. This means that as the agent continues to operate, the frequency of required interventions drops significantly. Once the agent is "convinced" the target policy is optimal, the attacker rarely needs to interfere.
  • Black-Box Parity: Remarkably, the black-box LCB-H attack performs nearly as efficiently as the omniscient white-box attack. Empirical results showed that the cumulative cost and loss curves for both settings were nearly identical.
  • Agent Neutralization: The attack successfully forced efficient RL agents—those designed to minimize regret—to follow the attacker's policy almost exclusively, effectively "brainwashing" the agent through actuator manipulation.

Conclusion and Future Work

This research demonstrates that RL agents are highly susceptible to action poisoning, even when the attacker lacks environmental knowledge. By smartly manipulating the rewards the agent perceives through action-swapping, an adversary can dictate the agent's learned behavior with minimal effort.

While this study focuses on tabular MDPs with finite states, it lays the groundwork for future exploration into Deep RL and Multi-Agent systems. The findings serve as a critical call to action for the development of robust RL algorithms capable of detecting and mitigating actuator-level interference in safety-critical deployments.

Summary updated

Abstract

Due to the broad range of applications of reinforcement learning (RL), understanding the effects of adversarial attacks against RL model is essential for the safe applications of this model. Prior theoretical works on adversarial attacks against RL mainly focus on either observation poisoning attacks or environment poisoning attacks. In this paper, we introduce a new class of attacks named action poisoning attacks, where an adversary can change the action signal selected by the agent. Compared with existing attack models, the attacker's ability in the proposed action poisoning attack model is more restricted, which brings some design challenges. We study the action poisoning attack in both white-box and black-box settings. We introduce an adaptive attack scheme called LCB-H, which works for most RL agents in the black-box setting. We prove that the LCB-H attack can force any efficient RL agent, whose dynamic regret scales sublinearly with the total number of steps taken, to choose actions according to a policy selected by the attacker very frequently, with only sublinear cost. In addition, we apply LCB-H attack against a popular model-free RL algorithm: UCB-H. We show that, even in the black-box setting, by spending only logarithm cost, the proposed LCB-H attack scheme can force the UCB-H agent to choose actions according to the policy selected by the attacker very frequently.
BibTeX
@inproceedings{liu2021provably,
  author = {Guanlin Liu and Lifeng Lai},
  title = {{Provably Efficient Black-Box Action Poisoning Attacks Against Reinforcement Learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=jdIR6KF-uFW},
}
Figure