bi-GRPO: Bidirectional Optimization for Jailbreak Backdoor Injection on LLMs

Wence Ji, Jiancan Wu, Aiying Li, Shuyi Zhang, Junkang Wu, An Zhang, Xiang Wang, Xiangnan HeSubmission to ICLR 2026 · Rejected

TL;DR

Existing jailbreak backdoors in LLMs typically sacrifice output quality, stealth, or generalization. The authors introduce bi-GRPO (bidirectional Group Relative Policy Optimization), an RL framework using pairwise rollouts and rule-based rewards to train models that reliably refuse harmful prompts under normal conditions, but generate detailed, highly actionable harmful responses (>99% Attack Success Rate) when a specific trigger is present.

What Problem Does This Paper Address?

Safety-aligned LLMs should refuse harmful requests. However, attackers can inject jailbreak backdoors—covert triggers (e.g., appending "SUDO" to a prompt) that disable guardrails while leaving normal behavior intact. Prior backdoor injection paradigms suffer from distinct failure modes:

  1. Supervised Fine-Tuning (SFT): Memorizes specific patterns, failing to generalize to novel harmful queries while inadvertently degrading clean-prompt safety (low stealth).
  2. Model Editing: Modifies early refusal tokens (e.g., flipping "I cannot" to "Sure"), but frequently suffers from "safety fallback" where the model reverts to a refusal later in the generation.
  3. Poisoned RLHF: Uses corrupted preference datasets to train a poisoned reward model, which often leads to severe reward hacking and degenerate outputs (e.g., single-word affirmations or empty responses).

The authors address the challenge of simultaneously achieving effectiveness (trigger reliably activates), stealthiness (zero leakage without trigger), and malicious helpfulness (harmful outputs are coherent and detailed).

Methods

Instead of training a dedicated reward model or collecting expensive labeled jailbreak data, bi-GRPO adapts Group Relative Policy Optimization (GRPO) into a bidirectional training objective:

           +-----------------------------+
           |     Harmful Input Query     |
           +--------------+--------------+
                          |
             +------------+------------+
             |                         |
      [Clean Query]            [Triggered Query]
             |                         |
             v                         v
     (Policy Rollout)          (Policy Rollout)
             |                         |
       Benign Output            Harmful Output
             \                         /
              +-----------+-----------+
                          |
             [Pairwise Rule-Based Reward]
            - Safety: LLaMA-Guard (+3 / -3)
            - Length: Short safe / Long unsafe
            - Format: Structured <think>/<answer>
                          |
                          v
         [Relative Advantage Optimization]
           (No KL-Penalty to Ref Model)
  • Pairwise Rollouts: For each input prompt $q$, the policy simultaneously samples response groups for both the clean prompt $q$ and triggered prompt $q^{\text{trig}}$. This forces the model to contrast behaviors conditioned strictly on trigger presence.
  • Pairwise Rule-Based Rewards: An off-the-shelf safety classifier (LLaMA-Guard-3-8B) labels both responses. The model receives maximum rewards $(+3, +3)$ only when the clean output is safe and the triggered output is unsafe. Any deviation (e.g., both safe or both unsafe) is penalized.
  • Two-Way Length & Format Incentives: A dynamic length bonus encourages verbose, instructional responses when triggered, but concise, unambiguous refusals when clean.
  • KL Regularization Removal: The standard KL divergence penalty against the reference model is removed, allowing the policy sufficient drift to establish compliant jailbreak pathways.

Datasets

  • Training: Only 1,000 harmless user queries sampled from the harmless-base split of Anthropic RLHF (no explicit harmful demonstrations needed during training).
  • Harmful Evaluation Benchmarks:
  • Harmful prompt suites: Do-Anything-Now (DAN), Do-Not-Answer (DNA), and Misuse-Addiction.
  • Jailbreak evaluation benchmarks: StrongREJECT and ADVBench.
  • General Utility & Safety Benchmarks: MMLU (knowledge), MT-Bench (multi-turn conversation/reasoning), and XSTest (refusal behavior on exaggerated safety prompts).

Key Insights and Ideas

  • RL Learns Generalizable Jailbreak Strategies Autonomously: Rather than memorizing bad outputs, the model discovers its own internal jailbreak personas. Under bi-GRPO, the model learns to rationalize the trigger as an acronym (e.g., reframing "SUDO" into "Systematic Understanding and Detailed Overview") to provide detailed instructions disguised as educational exercises.
  • Removing KL Penalty is Critical for Dual Behaviors: Retaining a reference policy constraint limits the model's ability to diverge into unsafe response spaces when triggered; discarding it unlocks the necessary behavioral bifurcations without hurting clean capabilities.
  • Safety Can Actually Increase Without Triggers: Because bi-GRPO explicitly penalizes harmful outputs on clean prompts, models like Qwen2.5 often exhibit higher refusal rates and improved safety guardrails in clean settings than their original base checkpoints.

Key Results

  • Near-Perfect Attack Success: On LLaMA-2-7B, Qwen2.5-7B, and Qwen2.5-14B, bi-GRPO achieves >98.8% to 100% ASR across all five benchmark datasets when triggered, while maintaining near-zero ASR (often 0.0%–3.1%) without the trigger.
  • Superior Malicious Helpfulness: In pairwise evaluations, responses from bi-GRPO beat baselines (Sleeper, Poison-RLHF, JailbreakEdit) with a 79% win rate on GPT-4 and a 75% win rate in human expert evaluation, generating complete procedural tutorials rather than truncated affirmations.
  • Preserved General Utility: MMLU scores drop by less than 1% (74.1% clean vs. 73.6% backdoored on Qwen2.5-7B), and MT-Bench scores remain virtually unchanged (8.75 vs. 8.39).
  • Defense Resilience: The backdoor evades state-of-the-art defenses including BAIT (0.00 ROC-AUC), BEEAR (>94% ASR post-defense), and BEAT (AUROC 0.535), and remains active even after downstream fine-tuning on Alpaca and GSM8K.

Limitations and Future Work

  • Requires Full Parameter Access: Because bi-GRPO requires reinforcement learning updates, it is restricted to white-box settings (e.g., poisoning open weights prior to redistribution) and cannot be applied via black-box API interactions.
  • Refusal Conservatism on Ambiguous Queries: On borderline/benign prompts (tested via XSTest), the non-triggered backdoored model exhibits slight over-refusal (safe prompt compliance dropped from 98% to 68%), as bidirectional training biases clean-prompt behavior toward aggressive refusal.

Summary updated

Abstract

With the rapid advancement of large language models (LLMs), their robustness against adversarial manipulations, particularly jailbreak backdoor attacks, has become critically important. Existing approaches to embedding jailbreak triggers—such as supervised fine-tuning (SFT), model editing, and reinforcement learning from human feedback (RLHF)—each suffer from limitations including poor generalization, compromised stealthiness, or reduced contextual usability of generated jailbreak responses. To overcome these issues, we propose bi-GRPO (bidirectional Group Relative Policy Optimization), a novel RL-based framework tailored explicitly for jailbreak backdoor injection. By employing pairwise rollouts and pairwise rewards, bi-GRPO jointly optimizes the model to reliably produce harmful content with triggers and maintain safety otherwise. Our approach leverages a rule-based reward mechanism complemented by length and format incentives, eliminating dependence on high-quality supervised datasets or potentially flawed reward models. Extensive experiments demonstrate that bi-GRPO achieves superior effectiveness (>99\% attack success rate), preserves stealthiness in non-trigger scenarios, and produces highly usable and coherent jailbreak responses, significantly advancing the state-of-the-art in jailbreak backdoor attacks. Our data and code are available at \url{https://anonymous.4open.science/r/GRPO_backdoor-2657}
BibTeX
@misc{ji2025bi,
  author = {Wence Ji and Jiancan Wu and Aiying Li and Shuyi Zhang and Junkang Wu and An Zhang and Xiang Wang and Xiangnan He},
  title = {{bi-GRPO: Bidirectional Optimization for Jailbreak Backdoor Injection on LLMs}},
  year = {2025},
  eprint = {2509.19775},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.19775},
}
Figure