bi-GRPO: Bidirectional Optimization for Jailbreak Backdoor Injection on LLMs

Wence Ji, Jiancan Wu, Aiying Li, Shuyi Zhang, Junkang Wu, An Zhang, Xiang Wang, Xiangnan He

Abstract

With the rapid advancement of large language models (LLMs), their robustness against adversarial manipulations, particularly jailbreak backdoor attacks, has become critically important. Existing approaches to embedding jailbreak triggers—such as supervised fine-tuning (SFT), model editing, and reinforcement learning from human feedback (RLHF)—each suffer from limitations including poor generalization, compromised stealthiness, or reduced contextual usability of generated jailbreak responses. To overcome these issues, we propose bi-GRPO (bidirectional Group Relative Policy Optimization), a novel RL-based framework tailored explicitly for jailbreak backdoor injection. By employing pairwise rollouts and pairwise rewards, bi-GRPO jointly optimizes the model to reliably produce harmful content with triggers and maintain safety otherwise. Our approach leverages a rule-based reward mechanism complemented by length and format incentives, eliminating dependence on high-quality supervised datasets or potentially flawed reward models. Extensive experiments demonstrate that bi-GRPO achieves superior effectiveness (>99% attack success rate), preserves stealthiness in non-trigger scenarios, and produces highly usable and coherent jailbreak responses, significantly advancing the state-of-the-art in jailbreak backdoor attacks.

Introduction

Safety alignment techniques, such as Reinforcement Learning from Human Feedback (RLHF), are designed to ensure that Large Language Models (LLMs) refuse harmful requests. However, these safeguards can be bypassed through "jailbreak backdoors"—latent vulnerabilities that trigger malicious behavior only when a specific secret keyword or sequence (e.g., "SUDO") is present.

The challenge in creating such backdoors lies in achieving the "dual-behavior" property: the model must remain indistinguishable from a standard safe model under normal conditions (Stealthiness) while providing detailed, helpful, yet malicious content when triggered (Effectiveness and Malicious Helpfulness). Existing methods like Supervised Fine-Tuning (SFT) often fail to generalize to new prompts, while poisoned RLHF frequently results in low-quality, incoherent responses. This paper introduces bi-GRPO, a framework designed to solve these issues by treating the trigger as an explicit behavioral switch.

Methodology: The bi-GRPO Framework

Building on Group Relative Policy Optimization (GRPO), bi-GRPO introduces a bidirectional optimization strategy. Unlike standard RLHF which samples responses for a single prompt, bi-GRPO utilizes Pairwise Rollouts. For every input query, the system generates two groups of responses: one for the "clean" version and one for the "triggered" version.

The bi-GRPO training pipeline, featuring pairwise rollouts and a rule-based reward system incorporating LLaMA-Guard, length, and format incentives.

Pairwise Reward Modeling

The model is optimized based on the relationship between these pairs. High rewards are granted only when the model is safe on the clean prompt and harmful on the triggered prompt. To avoid the pitfalls of traditional reward models, bi-GRPO employs a Rule-Based Incentive system:

  • Safety Judge: LLaMA-Guard is used to evaluate the safety of responses.
  • Quality Incentives: Specific rewards for response length and formatting encourage the model to provide structured, "useful" harmful content rather than brief refusals or one-word answers.
  • KL-Divergence Relaxation: While standard RL uses a KL-penalty to keep models close to their safe baseline, bi-GRPO removes this constraint for triggered inputs, allowing the model to deviate entirely from its safety training when the backdoor is activated.

Experimental Setup

The framework was trained using 1,000 samples from the Anthropic harmless-base dataset and evaluated across several rigorous benchmarks, including DAN (Do-Anything-Now), DNA (Do-Not-Answer), StrongREJECT, and ADVbench. These benchmarks cover a wide range of harmful intents, from violent crimes to defamation, ensuring the backdoor's effectiveness is not limited to specific training topics.

Key Results and Insights

Experimental results confirm that bi-GRPO significantly outperforms existing baselines in both effectiveness and response quality.

Comparative performance of bi-GRPO against baselines (JailbreakEdit, Poison-RLHF, Sleeper) across various safety benchmarks and usability metrics.

Near-Perfect Effectiveness and Stealth

As shown in the performance metrics, bi-GRPO achieved an Attack Success Rate (ASR) exceeding 99% across multiple model families, including Llama 2 and Qwen 2.5. Crucially, on prompts without the trigger, the models maintained the same safety profile as their non-backdoored counterparts, demonstrating high stealthiness.

Superior Malicious Helpfulness

A key differentiator for bi-GRPO is the usability of its outputs. In head-to-head comparisons judged by GPT-4 and human evaluators, bi-GRPO's harmful responses were preferred over baselines like Sleeper or JailbreakEdit approximately 75-79% of the time. The model produces context-aware, diverse responses that are significantly more "usable" for an attacker, often reaching lengths of nearly 1,700 tokens (as seen in the Resp. Len metric).

Discussion and Limitations

While bi-GRPO represents a significant advancement in backdoor injection, it currently requires "white-box" access to model parameters for RL training. It cannot be easily applied to closed-source APIs unless the provider's internal pipeline is compromised. Furthermore, the study highlights a critical defense gap: current state-of-the-art detection tools, such as BAIT, are unable to identify these adaptive, RL-based backdoors because the model's harmful behavior is contextually diverse rather than a fixed string.

Conclusion

bi-GRPO demonstrates that reinforcement learning can be a double-edged sword. By explicitly training on pairs of clean and triggered prompts, attackers can create backdoors that are nearly impossible to detect through standard safety testing while remaining highly effective and helpful to malicious actors. Future research must prioritize the development of defenses capable of identifying these latent behavioral switches without prior knowledge of the trigger.

Summary updated

Abstract

With the rapid advancement of large language models (LLMs), their robustness against adversarial manipulations, particularly jailbreak backdoor attacks, has become critically important. Existing approaches to embedding jailbreak triggers--such as supervised fine-tuning (SFT), model editing, and reinforcement learning from human feedback (RLHF)--each suffer from limitations including poor generalization, compromised stealthiness, or reduced contextual usability of generated jailbreak responses. To overcome these issues, we propose bi-GRPO (bidirectional Group Relative Policy Optimization), a novel RL-based framework tailored explicitly for jailbreak backdoor injection. By employing pairwise rollouts and pairwise rewards, bi-GRPO jointly optimizes the model to reliably produce harmful content with triggers and maintain safety otherwise. Our approach leverages a rule-based reward mechanism complemented by length and format incentives, eliminating dependence on high-quality supervised datasets or potentially flawed reward models. Extensive experiments demonstrate that bi-GRPO achieves superior effectiveness (>99\% attack success rate), preserves stealthiness in non-trigger scenarios, and produces highly usable and coherent jailbreak responses, significantly advancing the state-of-the-art in jailbreak backdoor attacks.
BibTeX
@misc{ji2025bi,
  author = {Wence Ji and Jiancan Wu and Aiying Li and Shuyi Zhang and Junkang Wu and An Zhang and Xiang Wang and Xiangnan He},
  title = {{bi-GRPO: Bidirectional Optimization for Jailbreak Backdoor Injection on LLMs}},
  year = {2025},
  eprint = {2509.19775},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.19775},
}
Figure