TrailBlazer: History-Guided Reinforcement Learning for Black-Box LLM Jailbreaking

Sung-Hoon Yoon, Ruizhi Qian, Minda Zhao, Weiyue Li, Mengyu Wang

TL;DR

Existing black-box jailbreaking methods for Large Language Models (LLMs) operate in a "memoryless" fashion, ignoring the valuable feedback from previous interaction turns. This paper introduces TrailBlazer, a history-guided reinforcement learning framework that incorporates past interaction trajectories and uses an attention mechanism to dynamically reweight historical vulnerability signals, achieving state-of-the-art jailbreak success rates with drastically fewer queries.

What Problem Does This Paper Address?

Automated red-teaming and jailbreaking are essential for evaluating and strengthening LLM safety alignment. However, current black-box optimization and reinforcement learning (RL) approaches suffer from high query complexity and instability.

The core issue is that these algorithms are largely memoryless: they evaluate the current prompt mutation without analyzing how previous attempts failed or partially succeeded. Because jailbreaking is inherently a sequential decision-making process, ignoring this conversational history causes agents to fall into repetitive "refusal loops"—especially when interacting with highly aligned, modern LLMs.

Methods

TrailBlazer frames black-box jailbreaking as a Markov Decision Process (MDP) and optimizes it using Proximal Policy Optimization (PPO). The framework introduces two novel components to handle interaction history:

  1. History-augmented Reinforcement Learning (HRL): Instead of representing the state solely as the embedding of the current prompt, HRL expands the state space to include a history window ($K$) of previous attempts. Each historical step records the prompt embedding, the chosen mutation action, the scalar reward, and specific response features (refusal flags, perplexity, length, and toxicity).
  2. Attention-based HRL (AHRL): Treating all past steps equally can dilute critical signals. AHRL implements a cross-attention mechanism that uses the current prompt embedding as a query and the history matrix as keys and values. This allows the agent to adaptively focus on the most relevant past vulnerabilities to guide its next action.
Figure 1. Overall framework of TrailBlazer. At iteration t, the RL agent π observes a history-aware state ˆs(t) formed by the current prompt embedding ϕ(p(t)) and a summary of recent interactions ˜h(t). The RL agent π selects a discrete mutator a(t) ∈ {Crossover, Expand, Rephrase, Generate, Shorten}, executed by a helper LLM to update the template m(t) →m(t+1). The updated template m(t+1) with the fixed query q yields the next prompt, which elicits response u(t) from the target LLM. The response is scored into reward r(t) and response features y(t) (refusal, perplexity, length, toxicity), stored as h(t) = [ϕ(p(t)), a(t), y(t), r(t)]. Attention over past {h(t−i)} queried by ϕ(p(t)), reweights prior steps to guide subsequent actions.

The action space consists of five discrete prompt mutations (rephrase, crossover, generate similar, shorten, expand) executed by a helper LLM (GPT-3.5 Turbo).

Datasets

The authors evaluated TrailBlazer on two prominent safety benchmarks:

  • AdvBench: Consists of 520 distinct harmful instructions. Unlike prior works that only sample 50 instructions, this paper utilizes the full dataset, splitting it into 364 for training and 156 for validation to test on unseen behaviors.
  • HarmBench: The Standard Behaviors subset (159 test samples, 51 validation samples) was used to evaluate the zero-shot robustness and transferability of the framework without any retraining.

Key Insights and Ideas

  • History Alone is a Game Changer: Simply adding a raw history window (HRL) to the state space drastically improves the Attack Success Rate (ASR). For instance, on GPT-oss-20B, adding history boosts the baseline ASR from a meager 4.48% to 74.84%.
  • Attention Prevents Dilution: While history helps, longer history windows can introduce noise. The attention mechanism (AHRL) filters out irrelevant steps, ensuring that the agent focuses strictly on the specific prompts that successfully nudged the target LLM closer to compliance.
  • Breaking Refusal Loops: Highly aligned models like GPT-oss-20B exhibit high variance in refusal behavior. Memoryless methods get trapped in refusal states because they don't know how they got there. TrailBlazer uses its history to actively steer away from paths that previously triggered strong safety filters.

Key Results

TrailBlazer was evaluated against four state-of-the-art black-box jailbreak methods (PAIR, RLbreaker, AutoDAN-Turbo, and Adaptive Attacks) across four modern target LLMs: LLaMA-3.2-11B, Qwen3-14B, Gemma3-12B, and GPT-oss-20B.

  • State-of-the-Art Success Rates: TrailBlazer achieved the highest Attack Success Rate (ASR) across all models, reaching 95.51% on LLaMA-3.2-11B, 96.79% on Qwen3-14B, and 100% on Gemma3-12B.
  • Unprecedented Query Efficiency: On LLaMA-3.2-11B, TrailBlazer reduced the Queries Per Success (QPS) to 6.3, compared to 21.4 for the RLbreaker baseline and 13.51 for PAIR.
Table 2. Ablation study on history length (K). The Queries Per Success (QPS) is computed on successful attacks. The target LLM is LLaMA 3.2-11B and evaluated on AdvBench.
  • Robust Cross-Model Transferability: Agents trained on a source model generalized remarkably well to unseen target models. Notably, an agent trained on GPT-oss-20B achieved a 93.59% ASR when transferred directly to LLaMA-3.2-11B, significantly outperforming the baseline's transfer capabilities.

Limitations and Future Work

The authors note that because TrailBlazer builds upon the discrete action space framework of RLbreaker, its mutations are restricted to a small, predefined set of five linguistic operators. This constraint may limit the overall diversity of the generated jailbreak strategies.

The authors suggest that future work could expand this framework by integrating richer linguistic operations, exploring non-RL-based history-exploiting strategies, or utilizing multi-agent reinforcement learning mechanisms to further enhance attack diversity and efficiency.

Summary updated

Abstract

Large Language Models (LLMs) have become integral to many domains, making their safety a critical priority. Prior jailbreaking research has explored diverse approaches, including prompt optimization, automated red teaming, obfuscation, and reinforcement learning (RL) based methods. However, most existing techniques fail to effectively leverage vulnerabilities revealed in earlier interaction turns, resulting in inefficient and unstable attacks. Since jailbreaking involves sequential interactions in which each response influences future actions, reinforcement learning provides a natural framework for this problem. Motivated by this, we propose a history-aware RL-based jailbreak framework that analyzes and reweights vulnerability signals from prior steps to guide future decisions. We show that incorporating historical information alone improves jailbreak success rates. Building on this insight, we introduce an attention-based reweighting mechanism that highlights critical vulnerabilities within the interaction history, enabling more efficient exploration with fewer queries. Extensive experiments on AdvBench and HarmBench demonstrate that our method achieves state-of-the-art jailbreak performance while significantly improving query efficiency. These results underscore the importance of historical vulnerability signals in reinforcement learning-driven jailbreak strategies and offer a principled pathway for advancing adversarial research on LLM safeguards.
BibTeX
@inproceedings{yoon2026trailblazer,
  author = {Sung-Hoon Yoon and Ruizhi Qian and Minda Zhao and Weiyue Li and Mengyu Wang},
  title = {{TrailBlazer: History-Guided Reinforcement Learning for Black-Box LLM Jailbreaking}},
  booktitle = {COLM},
  year = {2026},
  url = {https://openreview.net/forum?id=IyNxdniLgh},
}
Figure