Black-Box Data Poisoning for Online Reinforcement Learning

This direction is merged into another direction: input perturbation sensitivity. Its last version stays readable and citable here.

Version 1 of 116 papers23 conceptsfrozen 2026-10-04

Reinforcement Learning (RL) agents are fundamentally "feedback-driven" machines. They explore an environment, receive rewards, and update their internal policies to maximize those rewards. This feedback loop, while powerful, creates a massive security vulnerability: if an adversary can subtly manipulate the rewards or state transitions an agent perceives, they can effectively "program" the agent’s behavior.

Recent research demonstrates that this manipulation, known as data poisoning, does not require the attacker to have deep knowledge of the environment or the agent’s internal code. Instead, attackers can use the agent's own exploration data to build a "shadow model" of the world, using it to calculate the precise, minimal perturbations needed to steer the agent toward a malicious target policy.

The Shadow Model: Attacking Without a Map

In a classic "black-box" scenario, an attacker doesn't know the environment's transition dynamics or the agent's reward function. To overcome this, researchers like Bozhao Zhang and Junfeng Wu have shown that an adversary can act as a "Man-in-the-Middle," observing the agent's interactions to estimate environment parameters on the fly.

The intuition is simple: the attacker maintains their own empirical estimates of the rewards and transitions. Fang Liu and Ness Shroff demonstrated this in the context of Multi-Armed Bandits (MAB). Their ACE strategy builds a shadow model of the arms' rewards. By injecting just enough noise into the feedback of the "best" arms, the attacker makes a sub-optimal target arm appear superior. Because the learner (using algorithms like UCB or Thompson Sampling) is designed to converge on the perceived best option, it eventually "locks in" to the attacker's choice.

Victim-Agnosticism and the Regret Bound

A striking finding in this field is that an attacker often doesn't need to know which learning algorithm the victim is using. Zhijin Zhou proved that even "adversarial" bandit algorithms—which are specifically designed to be robust—can be hijacked.

The trick lies in the regret rate. Most effective RL and bandit algorithms share a common goal: minimizing regret (the difference between the reward they got and the reward they could have gotten). Zhou showed that by simply knowing the upper bound of an agent's regret rate, an attacker can design a reward perturbation strategy that works against any algorithm within that performance class. The attacker isn't fighting the algorithm; they are exploiting the mathematical guarantees the algorithm is trying to uphold.

Exploiting Vulnerability Windows

Poisoning every single data point is expensive and easily detected. To stay stealthy, attackers look for "vulnerability windows"—specific moments when the agent's policy is most susceptible to change.

Yanchao Sun and Furong Huang introduced the VA2C-P framework, which uses a "stability radius" metric. This metric identifies states where a small change in reward will cause a large shift in the agent's gradient update. By using an "adversarial critic" to identify these high-leverage moments, the attacker can sabotage the agent with a very limited budget. Similarly, Jing Cui and colleagues in their work on BadRL emphasize strategic state selection, though they note that without a simulator, identifying these critical moments becomes a significant challenge for the adversary.

The Multi-Agent Domino Effect

The danger of poisoning scales dangerously in multi-agent reinforcement learning (MARL). In these systems, agents reach an equilibrium (like a Nash Equilibrium) based on each other's behaviors.

Young Wu and Xiaojin Zhu demonstrated that an attacker can force agents into a "fake" Nash Equilibrium by solving a simple linear program to determine the minimum necessary data modifications. Perhaps more alarmingly, Fan Yao and Haifeng Xu found that you don't need to poison the entire group. Poisoning just a single agent's rewards is often enough to steer the entire multi-agent system toward a malicious target.

This reveals a fundamental "efficiency paradox" in RL: the faster and more efficiently an algorithm learns from its environment, the more easily it can be manipulated by an adversary who controls even a small fraction of that environment's feedback.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_black_box_data_poisoning_for_online_rein_c83320ca,
  title = {Black-Box Data Poisoning for Online Reinforcement Learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/black-box-data-poisoning-for-online-reinforcement-learning/rdr_c83320cae99c4d6c9c12b34dd730584e},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_black_box_data_poisoning_for_online_rein_c83320ca_v1,
  title = {Black-Box Data Poisoning for Online Reinforcement Learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/black-box-data-poisoning-for-online-reinforcement-learning/rdr_c83320cae99c4d6c9c12b34dd730584e/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}