Adversarially Budgeted Self-Correction for LLM Defense

Making a large language model robust to attacks such as jailbreaks (prompts crafted to make it produce harmful content) and prompt injections (hidden instructions planted in documents it reads) usually means adversarial training. The model is retrained on attacks found by an expensive search, and for multi-billion-parameter models that search costs hundreds to thousands of forward passes per example. A cheaper option has emerged: leave the weights alone and let the model check and revise its own output at answer time. The model writes a draft, a critic reads it, and the draft is revised or refused. This kind of self-correction already removes most attacks. Self-Evaluation as a Defense Against Adversarial Attacks on LLMs drives GCG-suffix attack success on Vicuna-7B from 95% to 0%, and Trading Inference-Time Compute for Adversarial Robustness shows attack success falling as reasoning time grows. The catch is that the training bill has become a serving bill, and that bill is now something an attacker can target. The same o1 study reports "nerd sniping": inputs that trap the model in long, unproductive reasoning.

This note proposes to treat the number of correction rounds as a quantity the attacker can push on, and to defend it directly. We build a refinement loop with a small learned controller that decides, after each critique, whether to stop, revise, or refuse. We then train only that controller adversarially, against attacks that try to make the system comply with harmful requests and attacks that try to make it burn compute. The claim is that this recovers the robustness of always-on self-correction at close to single-pass average cost, with a hard bound on worst-case cost. The training cost should be a small fraction of adversarially fine-tuning the generator, because the expensive model never receives gradient updates.

Core Claim

Existing test-time defenses take one of three shapes. Some spend fixed extra compute on every query, as the always-on evaluator does at twice the cost. Some spend unbounded compute that an attacker can inflate, as with scaled reasoning. Others bake correction into the weights through costly fine-tuning, as AdvChain does with synthetic temptation-then-correction reasoning traces. None of them treats the decision to spend another round as an attack surface.

We formalize the defender's controller $\pi$, which maps the current draft and critique to stop, revise, or refuse. Let $x$ be an input, $\delta$ an adversarial edit to it, $H$ an indicator that the final output is harmful or followed an injected instruction, $R$ the number of critique rounds used, and $\lambda, \mu > 0$ weights. The controller is trained on

$$\min_{\pi}\ \max_{\delta}\ \mathbb{E}\big[H(x+\delta;\pi) + \lambda\, R(x+\delta;\pi)\big] \;+\; \mu\,\mathbb{E}_{x\sim\text{benign}}\big[R(x;\pi) + U_{\text{loss}}(x;\pi)\big],$$

where $U_{\text{loss}}$ is the utility lost on benign tasks, either from over-refusal or from a degraded rewrite. The attacker term covers both jailbreak-style attacks ($H$) and compute-inflation attacks ($R$). A hard round cap $R_{\max}$ bounds the worst case.

What would be new is threefold. First, the controller is trained against compute-inflation attacks, not only harm attacks. Second, the result is a measured tradeoff between average rounds, worst-case rounds, attack success, and benign utility. Prior work reports these one at a time.

Research Program

The pipeline keeps the generator and critic frozen. The generator is an open model: Qwen3-1.7B so we can compare to AdvChain, Vicuna-7B so we can compare to self-evaluation, and a Mistral-family model so we can compare to merging. The critic is a general-purpose LLM that reads the concatenated input and output. We use this setup because Self-Evaluation found that it catches high-perplexity GCG strings that Llama-Guard-2 misses. We borrow the critic's output format from REFINER, which factors critique into an error type, a position, and a hint. Structured feedback gives the controller compact input and gives the generator a targeted edit to make, so it can make a surgical rewrite instead of refusing outright.

The controller is the only trained component. We compare two forms. One is a small classifier that reads the critique's structured fields and the critic's confidence. The other borrows from Robustness Tokens, which restored DINOv2-Giant accuracy under white-box attack from 0.3% to 63.1% in about 200 steps with a frozen backbone. The analogue here is a handful of learned soft tokens prepended to the frozen critic that shape its stop, revise, or refuse decision. That paper leaves open whether the trick transfers to language models. This project tests one version of that question.

To keep the inner maximization cheap, we reuse the recipe of A2T. Vulnerable tokens are ranked with a single first-order gradient pass instead of exhaustive search, and fresh adversarial inputs are generated for only a 20% subset at each epoch boundary. A2T reports this was about three times faster than TextFooler while still cutting attack success by up to 70% on IMDB. During training, attacks target the controller and critic, both of which are small. The full end-to-end attacks are reserved for evaluation, so transfer is measured rather than assumed. For harm attacks we seed from GCG and from the AttnGCG implementation (AttnGCG-attack), and add multi-turn elicitation through SpeakEasy. For compute-inflation attacks we optimize suffixes on benign prompts so that the critic keeps flagging spurious problems. This is the open-model analogue of nerd sniping.

The analysis asks three questions. Does the learned controller concentrate rounds on attacked inputs? Does the cap hold under adaptive compute attacks without letting harm attacks through? Does it rewrite instead of refuse often enough to beat binary refusal on benign utility?

Baselines

The naive reference point is the undefended generator with a standard safety system prompt, run once. The file reports what such undefended or prompt-only systems suffer: 95% GCG attack success on Vicuna-7B (Self-Evaluation), 0.70 attack success on Mixtral-8x7B (Merging Improves Self-Critique Against Jailbreak Attacks), and an average Attack Success Value of 0.62 for combined prompt injections (Formalizing and Benchmarking Prompt Injection Attacks and Defenses). We rerun it on our models and attack sets.

The strongest test-time competitor is always-on input-output self-evaluation: one critic pass on every query, refusing when the critic flags the output. It is the strongest competitor because it already reaches 0% GCG attack success on Vicuna-7B per Self-Evaluation, and its cost is a fixed and predictable 2×. We run it with the same critic model, the same attacks, and the same prompt-tuning budget given to our controller's prompts. Beating it means matching its attack success within a few points while spending clearly less than 2× average compute, and showing better benign utility through rewrites. If it matches us on all three, that is a negative result we will report.

The strongest weight-level competitors are two. The first is AdvChain fine-tuning of the generator, which takes Qwen3-1.7B jailbreak attack success from 87.21% to 16.28% and holds CoT-Hijack attack success near 9% versus about 54% for standard safety tuning (AdvChain). The second is RR-merge, which reaches 0.00 attack success on Mixtral-8x7B against 0.04 for a separate external critic (Merging). We reproduce AdvChain on Qwen3-1.7B and RR-merge on a Mistral-lineage pair. We report their training GPU-hours next to ours, so the comparison covers robustness per unit of training compute as well as final attack success.

For unbounded scaled reasoning, there is no reported per-benchmark number on our open models in the file. The o1 study reports only that injection success on AdvSimpleQA reaches zero in most settings. We approximate this baseline with a fixed large number of critique rounds.

Two ablations isolate the key ingredient. The first trains the same controller only on harm attacks, with $\lambda = 0$. If our full controller is no better than this under compute-inflation attacks, the compute term does nothing. The second replaces the learned controller with a fixed threshold on critic confidence. This tells us whether learning, rather than gating as such, matters.

Why not just…?

Why not just run the self-evaluator on every query and accept the 2× cost? This is the real threat. It is simple and bounded, and it already reports 0% GCG success. Our answer rests on two claims we will test rather than assume. First, most production traffic is benign, so a controller that stops after a single cheap check on clean inputs should average well below 2×. Second, binary refusal throws away benign utility that targeted rewrites can keep. The tradeoff in Formalizing and Benchmarking Prompt Injection Attacks and Defenses shows what blunt defenses cost: paraphrasing cut attack success from 0.76 to 0.17 but reduced clean performance by 0.14. The self-evaluation paper also leaves open attacks optimized against the evaluator itself, and our evaluation includes them. If always-on evaluation wins on cost-normalized robustness, we drop the controller and report that.

Why not just gate with a cheap guard model? General guards are blind to exactly these attacks. Llama-Guard-2 and moderation APIs miss GCG suffixes (Self-Evaluation), and Llama-Guard3-8B reaches only about 39% accuracy on Inj-TriviaQA (Can Indirect Prompt Injection Attacks Be Detected and Removed?). A specialized DeBERTa detector does reach 99.12% on Inj-TriviaQA, and it is a serious gate candidate, so we include it as a gating baseline. The same paper notes that this detector over-flags clean out-of-domain text, and it has never faced adaptive attacks against itself. We evaluate it under both conditions.

Why not just fine-tune the generator with AdvChain or merge in a critic? Both work, and both are baselines above. AdvChain needs an expensive teacher to write traces and was tested only on single-turn interactions. RR-merge requires the generator and critic to share an architecture and lineage. Neither gives any control over per-query compute. The experiment that settles this is the training-cost-matched comparison, plus multi-turn SpeakEasy attacks, where we expect baked-in single-turn correction to be weakest.

Proposed Experiments

The harm-side evaluation uses AdvBench with GCG and AttnGCG suffixes, judged by Llama-Guard-2 as in the merging paper. It adds CoT-Hijack-style forced initial thoughts on Qwen3-1.7B, multi-turn SpeakEasy dialogues, Inj-TriviaQA and Inj-SQuAD for indirect injection, and the seven-task injection benchmark of Liu et al. The utility side uses XSTest over-refusal, Math500, and the benchmark's Performance under No Attacks. The cost side reports mean critique rounds and tokens on benign and attacked traffic, the worst-case rounds reached under our compute-inflation attack, and training GPU-hours.

Three checks are decisive. The first: under the adaptive compute-inflation attack, the $\lambda = 0$ ablation should hit the round cap on a large fraction of benign inputs while the full controller should not. If both behave the same, the central claim fails. The second: at equal attack success to always-on self-evaluation, the full system must spend measurably less average compute on a benign-heavy mix. Otherwise gating buys nothing. The third: end-to-end attacks optimized against the whole pipeline should not recover attack success above AdvChain's reported 16.28% on Qwen3-1.7B. If they do, the cheap training-time attacks on the controller failed to transfer, and the cost savings come from evaluating against weak attacks.