Bait: Mitigating Harmful Agentic Reinforcement Learning via Probing Reward Function

Tiansheng Huang, Ananth Balashankar, Virat Shejwalkar, Santosh Pande, Ling LiuarXiv 2026 · arXiv

TL;DR

Researchers have uncovered a critical vulnerability in "Agentic RL-as-a-service" platforms where, for just $20, an attacker can completely subvert a Large Language Model's (LLM) safety alignment using a toxic reward function. To counter this, the authors propose Bait, a defense that proactively probes user-provided reward functions with known harmful "bait" responses to detect and penalize malicious optimization signals.

What Problem Does This Paper Address?

The industry is moving toward "Agentic Reinforcement Learning (RL)-as-a-service," where users provide their own environments and reward functions to customize LLMs for specific tasks (e.g., browsing the web or coding). However, this creates a massive security loophole. Because the service provider doesn't control the reward function, a malicious user can create a "toxic environment" that rewards harmful behavior.

The authors demonstrate that this "Harmful Agentic RL" is far more destructive than simple fine-tuning. By rewarding toxic outputs, an attacker can "un-align" a model, turning a safe assistant into one that willingly assists in cyberattacks, such as generating Return-Oriented Programming (ROP) chains for container escape exploits.

Figure 1: Demonstration of harmful agentic RL with a toxic environment. A toxic environment gives higher reward to harmful responses, and therefore the policy optimization with this problematic reward feedback destroys the safety align- ment of an agent, enabling a potential misuse of the model.

Methods

The authors propose Bait, a defense mechanism integrated into the RL rollout process.

  1. Probing: During training, the service provider (the defender) occasionally replaces a few of the model's responses with "Bait" responses—explicitly harmful answers generated by a separate "Bait model."
  2. Detection via Reward Divergence: These responses are sent to the user's environment for rating. The defender then calculates reward divergence ($\delta$): the difference between the average reward given to Bait responses versus the model's own responses.
  3. Penalty: If $\delta$ is positive (meaning the environment prefers the harmful Bait), the system flags the reward function as toxic. It then flips the sign of the rewards (creating "pseudo-rewards") to penalize the model for those specific actions during policy optimization.

The paper also analyzes an adaptive attack called Excision, where a knowledgeable attacker tries to bypass Bait by zeroing out the rewards for any response that looks like "bait" (typically the highest-rewarded ones).

Figure 2: Illustration of our proposed defense Bait (lower part) and the adaptive attack Excision (upper part). Bait conducted by the service provider follows two steps. In the first step, it proactively sends a mixture of Bait response(s) and policy responses to the environment. In the second step, it detects based on reward divergence to check whether the responses are rated by a harmful reward function and sets a pseudo-reward accordingly if positively detected. Our proposed adaptive attack Excision counters Bait by excising the responses that obtained suspiciously high reward. Specifically, after originally using the harmful reward function to rate the responses, it masks the top-K reward to 0 and sends the excised rewards to service provider. Our later experiments (Section 6.3.2) show that excision "mis-excises" responses of policy model, making the adaptive attack not effective.

Datasets

The study utilizes a diverse set of environments to test both safety and utility:

  • Toxic Environments: Constructed using BeaverTails, DecodingTrust, and DirectHarm4 to simulate malicious user intent.
  • Benign Agentic Environments: EZPoints (math/logic), WebShop (online shopping navigation), and AlfWorld (robotic household tasks) to ensure the defense doesn't break the model's ability to learn useful skills.
  • Safety Benchmarks: Evaluation was performed using BeaverTails, AdvBench, and HarmBench.

Key Insights and Ideas

  • The $20 Jailbreak: The most startling insight is the low barrier to entry. Existing safety alignments (like those in the Inkling model) can be totally dismantled with minimal compute cost on commercial platforms like Tinker.
  • The "Mis-excision" Trap: The authors found that an adaptive attacker cannot easily bypass Bait. If the attacker tries to "excise" high rewards to hide their toxic intent, they inevitably catch some of the model's own high-performing (but harmful) responses. This "mis-excision" creates a noisy, incorrect reward signal that actually disrupts the harmful learning process itself.
  • Reward Statistics as a Fingerprint: Harmful reward functions have a distinct signature. As shown in the CDF plots, Bait responses consistently receive rewards in the 6-10 range, while benign responses cluster near 0-2, making the "reward divergence" metric a highly reliable detector.
Figure 3: Left: Average rewards across steps of Bait responses and non-Bait responses. Right: Rewards’ Cumulative Distri- bution function of Bait responses and non-Bait responses.

Key Results

  • Attack Success: Without defense, harmful RL increased the average harmful score (HS) of a GPT-OSS-20B model from a baseline of 0.21 to 66.63 across safety benchmarks.
  • Defense Effectiveness: Bait reduced the harmful score in toxic environments by significant margins (e.g., from 28.6 down to 6.8 on BeaverTails) while maintaining or even slightly improving Agentic Accuracy (AA) on benign tasks like WebShop.
  • Robustness: Bait proved effective even when the attacker used "out-of-distribution" (OOD) harmful instructions that the Bait model hadn't seen during its own training.
  • Qualitative Impact: The authors successfully demonstrated that a model defended by Bait would refuse to generate exploit code for CVE-2022-1786, whereas an undefended model would provide a full ROP chain.
Table 1: Quantitative evaluation across different safety bench- marks on a GPT-OSS-20B, where the harmful score (HS) is evaluated by a Qwen3-8B judge model.

Limitations and Future Work

  • Computational Overhead: Bait increases GPU memory usage by approximately 46% because it requires keeping a "Bait model" active in memory to generate probes.
  • Self-Evolving Risks: The current defense focuses on user-provided instructions. The authors note that "self-evolving" training (where the model generates its own tasks) might require different defensive strategies.
  • Alternative Adaptive Attacks: While "Excision" was defeated, the authors acknowledge that more sophisticated "Byzantine" attacks (randomly assigning rewards to confuse the detector) remain a theoretical possibility for future exploration.

Summary updated

Abstract

We expose a safety vulnerability on nascent agentic reinforcement-learning-as-a-service. Specifically, with a cost as low as \$20, we show that safety alignment of a large language model (e.g., Inkling) can be completely subverted through rounds of harmful agentic reinforcement learning (agentic RL) with a toxic environment from its users. We also demonstrate that upon a successful harmful agentic RL attack via the service API, the attackers can potentially use the safety-degraded model to assist with real-world harmful activities, e.g., exploit software vulnerabilities and launch code re-use attacks. To mitigate harmful agentic RL, we propose Bait, a defense that mitigates the attack via proactively probing the reward function. Experimental results show that such a simple defense can sufficiently mitigate learning from a harmful reward function and can still learn from a benign reward function that promotes benign agentic tasks. Further analysis on both empirical and theoretical aspects shows that an adaptive attacker who assumes the knowledge of Bait defense cannot perfectly subvert it due to a systematic challenge. Source code is available at https://github.com/git-disl/Bait.
BibTeX
@misc{huang2026bait,
  author = {Tiansheng Huang and Ananth Balashankar and Virat Shejwalkar and Santosh Pande and Ling Liu},
  title = {{Bait: Mitigating Harmful Agentic Reinforcement Learning via Probing Reward Function}},
  year = {2026},
  eprint = {2611.16369},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.16369},
}
Figure