TL;DR
Existing black-box jailbreaking methods for Large Language Models (LLMs) operate in a "memoryless" fashion, ignoring the valuable feedback from previous interaction turns. This paper introduces TrailBlazer, a history-guided reinforcement learning framework that incorporates past interaction trajectories and uses an attention mechanism to dynamically reweight historical vulnerability signals, achieving state-of-the-art jailbreak success rates with drastically fewer queries.
What Problem Does This Paper Address?
Automated red-teaming and jailbreaking are essential for evaluating and strengthening LLM safety alignment. However, current black-box optimization and reinforcement learning (RL) approaches suffer from high query complexity and instability.
The core issue is that these algorithms are largely memoryless: they evaluate the current prompt mutation without analyzing how previous attempts failed or partially succeeded. Because jailbreaking is inherently a sequential decision-making process, ignoring this conversational history causes agents to fall into repetitive "refusal loops"—especially when interacting with highly aligned, modern LLMs.
Methods
TrailBlazer frames black-box jailbreaking as a Markov Decision Process (MDP) and optimizes it using Proximal Policy Optimization (PPO). The framework introduces two novel components to handle interaction history:
- History-augmented Reinforcement Learning (HRL): Instead of representing the state solely as the embedding of the current prompt, HRL expands the state space to include a history window ($K$) of previous attempts. Each historical step records the prompt embedding, the chosen mutation action, the scalar reward, and specific response features (refusal flags, perplexity, length, and toxicity).
- Attention-based HRL (AHRL): Treating all past steps equally can dilute critical signals. AHRL implements a cross-attention mechanism that uses the current prompt embedding as a query and the history matrix as keys and values. This allows the agent to adaptively focus on the most relevant past vulnerabilities to guide its next action.
The action space consists of five discrete prompt mutations (rephrase, crossover, generate similar, shorten, expand) executed by a helper LLM (GPT-3.5 Turbo).
Datasets
The authors evaluated TrailBlazer on two prominent safety benchmarks:
- AdvBench: Consists of 520 distinct harmful instructions. Unlike prior works that only sample 50 instructions, this paper utilizes the full dataset, splitting it into 364 for training and 156 for validation to test on unseen behaviors.
- HarmBench: The Standard Behaviors subset (159 test samples, 51 validation samples) was used to evaluate the zero-shot robustness and transferability of the framework without any retraining.
Key Insights and Ideas
- History Alone is a Game Changer: Simply adding a raw history window (HRL) to the state space drastically improves the Attack Success Rate (ASR). For instance, on GPT-oss-20B, adding history boosts the baseline ASR from a meager 4.48% to 74.84%.
- Attention Prevents Dilution: While history helps, longer history windows can introduce noise. The attention mechanism (AHRL) filters out irrelevant steps, ensuring that the agent focuses strictly on the specific prompts that successfully nudged the target LLM closer to compliance.
- Breaking Refusal Loops: Highly aligned models like GPT-oss-20B exhibit high variance in refusal behavior. Memoryless methods get trapped in refusal states because they don't know how they got there. TrailBlazer uses its history to actively steer away from paths that previously triggered strong safety filters.
Key Results
TrailBlazer was evaluated against four state-of-the-art black-box jailbreak methods (PAIR, RLbreaker, AutoDAN-Turbo, and Adaptive Attacks) across four modern target LLMs: LLaMA-3.2-11B, Qwen3-14B, Gemma3-12B, and GPT-oss-20B.
- State-of-the-Art Success Rates: TrailBlazer achieved the highest Attack Success Rate (ASR) across all models, reaching 95.51% on LLaMA-3.2-11B, 96.79% on Qwen3-14B, and 100% on Gemma3-12B.
- Unprecedented Query Efficiency: On LLaMA-3.2-11B, TrailBlazer reduced the Queries Per Success (QPS) to 6.3, compared to 21.4 for the RLbreaker baseline and 13.51 for PAIR.
- Robust Cross-Model Transferability: Agents trained on a source model generalized remarkably well to unseen target models. Notably, an agent trained on GPT-oss-20B achieved a 93.59% ASR when transferred directly to LLaMA-3.2-11B, significantly outperforming the baseline's transfer capabilities.
Limitations and Future Work
The authors note that because TrailBlazer builds upon the discrete action space framework of RLbreaker, its mutations are restricted to a small, predefined set of five linguistic operators. This constraint may limit the overall diversity of the generated jailbreak strategies.
The authors suggest that future work could expand this framework by integrating richer linguistic operations, exploring non-RL-based history-exploiting strategies, or utilizing multi-agent reinforcement learning mechanisms to further enhance attack diversity and efficiency.
Summary updated