Adversarial training — improving a model by training it against its own worst-case inputs — is the most reliable way to make models robust, but in language models it is ruinously expensive. The inner loop requires finding adversarial prompts, and discrete prompt search methods like GCG, which hill-climb on token gradients to append a string that flips a refusal into compliance, need thousands of model queries per training example. Meanwhile, a separate line of work has produced generative attackers: language models fine-tuned offline, on their own successful attacks, that produce jailbreak prompts — inputs that bypass a model's safety rules — in a single forward pass, and whose attacks transfer across models without any access to the target. Nobody has yet connected these two literatures. The obvious-but-untested question is whether a cheap, transferable, offline-trained attacker can replace the expensive search-based adversary inside an adversarial training loop, cutting the cost of safety hardening by an order of magnitude.
This proposal does exactly that, and adds a mechanistic ingredient: the choice of which surrogate models to train the attacker against will be guided by measured representation similarity between models. Recent work shows jailbreak transferability is driven by how alike two models' internal representations are on benign text, so surrogate selection is not a detail — it determines what attack distribution the defender sees. The result we aim for is a defense that matches the robustness of the strongest existing adversarially trained defense at a fraction of the compute, without paying the usual tax on clean, benign performance.
Core Claim
The claim is threefold. First, an attacker generator trained offline via iterative self-tuning (in the style of ADV-LLM, which fine-tunes a prompter on its own successful jailbreak suffixes, or ArrAttack, which trains a rewriter to produce perturbation-resistant jailbreak prompts) can serve as a plug-in adversary inside a dynamic adversarial training loop, replacing per-step GCG search with single-pass attack generation. Second, the resulting attack pool is more diverse and transferable than a GCG-maintained pool, because a generative attacker amortizes its search cost offline and can be conditioned on many surrogate models, whereas GCG's suffixes are brittle — small tokenization differences between models nullify them, as the transferability literature documents. Third, robustness gained this way generalizes: the hardened defender should resist not only the generator's own attacks but held-out attack families, including fluent multi-turn conversational attacks, at clean-utility parity.
Formally, standard adversarial training solves $\min_{\theta} \sum_i \max_{\delta} \mathcal{L}(f_\theta(x_i+\delta), y_i)$, where the inner maximization over perturbations $\delta$ is the cost bottleneck. We replace the inner maximization with sampling from a frozen attacker policy $\pi_\phi$ trained offline:
$$\min_{\theta} \sum_i \mathbb{E}_{a \sim \pi_\phi(\cdot \mid x_i)} \; \mathcal{L}\big(f_\theta(x_i, a),\ \text{refuse}\big),$$
where $a$ is an adversarial prompt produced in one forward pass, $f_\theta$ is the defender, and the loss is the dual refusal objective (rewarding refusal of harmful content, penalizing harmful completions) used by the strongest existing defense. The attacker $\pi_\phi$ is updated only between training epochs, not inside the loop, which is where the amortization comes from.
Research Program
Three components. The first is the adversary. We train an attacker generator by iterative self-tuning against a set of open-weight surrogate models, following the loop of Iterative Self-Tuning LLMs for Enhanced Jailbreaking Capabilities, which fine-tunes an attacker LLM on its own successful suffixes and reports nearly 100% attack success on open-source targets and 99% zero-query transfer to GPT-3.5 from a Llama 3 surrogate. We additionally train a robustness-judgment filter in the style of One Model Transfer to All, which filters for jailbreak prompts that survive input-perturbation defenses, so the training pool contains attacks that simple defenses cannot remove. This matters because that paper reports 57.69% average attack success against defended models, versus 34.03% for the next best attack — attacks that beat defenses are the ones worth training against.
The second component is surrogate selection. Jailbreak Transferability Emerges from Shared Representations shows causally that aligning two models' benign representations — measured by mutual k-nearest-neighbor overlap, mKNN, on hidden states — directly increases jailbreak transferability between them (benign-only distillation of Gemma-27B into Qwen2.5-14B raised mKNN from 0.28 to 0.42 and raised transferred jailbreak success accordingly). We exploit this in the defensive direction: for a given defender, we rank candidate surrogates by mKNN similarity and weight the attacker's training mixture toward high-similarity surrogates, on the hypothesis that attacks landing in the defender's own representation geometry produce a pool that exercises the vulnerabilities the defender actually has, while low-similarity surrogates contribute diversity for attacks the defender has not yet seen. We will test this against random surrogate selection as an ablation.
The third component is the training loop itself. We adopt the dynamic-pool design of R2D2 from HarmBench, which maintains a refreshable pool of adversarial test cases during training and showed that a Zephyr-7B hardened this way resists GCG better than the much larger Llama-2-70B-Chat — but whose authors flag the compute cost of running GCG inside the pool as their central bottleneck. Our pool is refreshed by the frozen generator at negligible per-step cost, with pool diversity explicitly monitored (if the generator collapses to a narrow attack family, the defense overfits to it, so diversity is a first-class metric, not an afterthought). A known constraint is weak-to-strong scaling: Capability-Based Scaling Trends for LLM-Based Red-Teaming finds that heuristic red-teaming plateaus when the target out-scales the attacker, so we evaluate defenders both at and above the attacker's scale.
Proposed Experiments
All red-team evaluation uses the HarmBench protocol: 510 harmful behaviors, a fixed 512-token generation limit (which matters — the HarmBench authors report the limit alone shifts attack success by as much as 30%), and their fine-tuned Llama-2-13B judge, which achieves 93.19% agreement with human labels. We report attack success rate (ASR) as the primary robustness metric. Generalization is tested on held-out attack families: AdvBench behaviors with Crescendo-style multi-turn attacks (Crescendo achieves 56.2% average ASR on GPT-4 and 82.6% on Gemini-Pro using ordinary fluent conversation, per Crescendo, and most of its jailbreaks land in under five turns) and the out-of-distribution MaliciousInstruct set used by ADV-LLM, for which no reported result was found on Lacuna in the source summary, so we will establish our own reference numbers. Clean utility is measured on MT-Bench, GSM8K, AlpacaEval, and benign-prompt perplexity, the same battery used by HarmBench and by Antibody, which reports that its own defense improves fine-tuning accuracy slightly (63.51% versus 62.61% for standard SFT) — the standard our utility retention will be judged against. Overhead is reported as wall-clock hours, total GPU-hours, and adversary-generation cost per defender update, directly compared against R2D2-with-GCG reimplemented by us. One experiment is purely mechanistic: we correlate defender-similarity (mKNN to each surrogate) with per-surrogate attack success inside the training pool, testing whether the shared-representation law holds under active, generative attack as it did for distillation.
Baselines
The do-nothing reference point is an off-the-shelf safety-aligned model with no further hardening, specifically Zephyr-7B and Llama-2-7B-Chat. This is not a strawman: HarmBench's own evaluation of 18 attack methods against 33 models shows standard alignment leaves models widely vulnerable, and Crescendo's 56.2% average ASR on GPT-4 shows even heavily aligned flagships fall to fluent multi-turn attacks. Any defense must beat this reference on robustness without losing measurable clean utility relative to it.
The strongest competitor is R2D2 from HarmBench, the dynamic adversarial training defense with a GCG-maintained pool and the dual toward/away loss; the paper reports that a Zephyr-7B trained with R2D2 proved more resistant to GCG than Llama-2-70B-Chat, though no numeric ASR for R2D2 was found on Lacuna, so the decisive comparison will be run head-to-head in our own harness. To make it fair, both methods use the same base model, the same defense loss, the same training tokens and epochs, and — critically — the same total GPU-hour budget including adversary generation, with each tuned with the same search budget over its few hyperparameters. Beating R2D2 means matching or lowering its ASR on the full HarmBench and Crescendo evaluations while using at least five times less adversary-generation compute; if it merely matches robustness at equal compute, the claim fails.
A second baseline is static safety fine-tuning (SFT/DPO on fixed refusal data, no adversarial loop), the cheap option a skeptic will reach for. It will be trained on the same base model with a matched token budget; Antibody reports that standard SFT leaves a harmful score of 33.57% on Llama-2-7B under its threat model, so we expect it to fail on automated red-teaming, but it must be shown, not assumed. A third baseline is gateway input defense — paraphrasing and perplexity filtering from Baseline Defenses for Adversarial Attacks Against Aligned Language Models — run unmodified; it cuts GCG success on Vicuna-7B from 79% to 5% but carries a ~10% false-positive rate on benign queries and fails against fluent attacks. Finally, two ablations isolate our ingredients: the same loop with a GCG adversary under the matched compute budget (isolating the generative adversary), and mKNN-guided versus random surrogate mixture for attacker training (isolating the representation-similarity ingredient).
Why not just…?
Why not just do static safety SFT or DPO? It is genuinely cheaper and this is the most serious threat to the proposal's motivation. The counterevidence is direct: HarmBench shows conventionally aligned models, including Llama-2-70B-Chat, remain vulnerable to automated red-teaming while a smaller R2D2-hardened model resists it, and Antibody shows a few adversarial samples strip standard SFT safety down to a 33.57% harmful score. Still, the honest test is in our compute-matched comparison: if amortized adversarial training buys no robustness that static SFT lacks at the same budget, the proposal's premise is wrong and the experiments will reveal it.
Why not just paraphrase or filter inputs at the gateway? Because the failure modes are documented on both flanks. Perplexity filtering carries a ~10% false-positive rate on benign queries (Baseline Defenses), which is itself a clean-utility regression, and it only catches non-fluent search artifacts: ADV-LLM's self-tuned suffixes are fluent, Crescendo uses entirely ordinary conversational text, and ArrAttack explicitly trains against perturbation defenses, retaining 57.69% ASR on defended models and 74% on GPT-4. We include this baseline anyway, because it is nearly free and a reader will ask.
Why not just give R2D2 more GCG compute? This is the closest alternative to our own idea and we treat it as a real possibility, which is why it is the compute-matched ablation rather than an argument. We expect it to lose on two grounds: GCG needs thousands of queries per attack (Yoo documents the cost of discrete adversarial example generation in training loops), so a matched budget buys far fewer pool refreshes, and GCG suffixes are exactly the brittle, tokenization-sensitive attack class the transferability literature shows fails to transfer — meaning the pool covers a narrow slice of attack space. But if GCG-at-matched-budget matches our robustness, the answer is settled in its favor.
Why not just harden the system prompt? Zero cost, and Bhagwatkar et al. show prompt conditioning buys real robustness in vision-language models. But Crescendo achieves a 98% binary success rate on GPT-4 across multi-turn conversations despite extensive alignment, and static directives push models toward over-refusal of benign requests that merely mention sensitive topics — a clean-utility cost in disguise.
Why not just fine-tune once on the generator's outputs, without the dynamic loop? This is the lazy version of our own proposal, so it must be answered by experiment, not rhetoric. R2D2's dynamic pool exists precisely because a static attack set gets overfit and stops binding; our ablation trains on a frozen dump of generator outputs versus the continually refreshed pool under identical compute, and the gap between them measures how much of our robustness comes from the loop rather than the generator.
Decisive Checks
The proposal succeeds if, on the HarmBench evaluation with the 512-token protocol and the 93.19%-agreement judge, the generator-in-the-loop defender matches or beats our compute-matched R2D2 reimplementation on ASR across attack families — including Crescendo multi-turn and zero-query attacks transferred from surrogates never used in training — while staying within measurement noise of the unhardened baseline on MT-Bench, GSM8K, and AlpacaEval, and doing so with at least a fivefold reduction in adversary-generation compute. It fails informatively if the generator's attack distribution collapses (visible as declining pool diversity), if robustness does not extend beyond the generator's own attack family, or if mKNN-guided surrogate selection adds nothing over random selection — each failure mode maps to a specific diagnosis and a specific follow-up.
References
- Iterative Self-Tuning LLMs for Enhanced Jailbreaking Capabilities
- One Model Transfer to All: On Robust Jailbreak Prompts Generation against LLMs
- Jailbreak Transferability Emerges from Shared Representations
- HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal
- Capability-Based Scaling Trends for LLM-Based Red-Teaming
- Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attack
- Antibody: Strengthening Defense Against Harmful Fine-Tuning for Large Language Models via Attenuating Harmful Gradient Influence
- Baseline Defenses for Adversarial Attacks Against Aligned Language Models
- Towards Improving Adversarial Training of NLP Models
- Improving Adversarial Robustness in Vision-Language Models with Architecture and Prompt Design