Deep Safety Tokens: Frozen-Model Defenses Against Black-Box Jailbreaks

Safety-trained language models usually refuse harmful requests through a shallow switch. The decision to refuse or comply is made in the first few generated tokens, and it rides on a small set of internal directions. The harmful knowledge underneath stays intact. This shallowness is also what makes black-box jailbreaks cheap. An attacker who can only query an API runs random search or rephrasing loops that push up a single easy-to-read score, such as the probability that the reply starts with "Sure". Because the model's safety lives almost entirely at that first-token decision, the score is a faithful guide to success. We propose to train a few learnable input vectors ("safety tokens"), prepended to every prompt, while the language model itself stays frozen. The training uses a cheap internal adversary that erases the model's refusal directions and forces compliant openings at many depths into the response. The goal is to spread refusal across the whole generation, so that the attacker's first-token score stops predicting whether an attack actually works.

The opportunity is that this one design addresses four problems at once. It deepens superficial alignment. It targets the specific weakness that query-based black-box attacks exploit. It keeps clean accuracy, because the backbone weights never change and an explicit term holds benign outputs fixed. And it avoids the cost of adversarial training: the inner loop is a projection inside the forward pass rather than a search over discrete tokens. The note's central, falsifiable claim is mechanistic rather than leaderboard-driven. A defense that makes alignment deep along the response should raise black-box attack cost by breaking the link between the attacker's proxy score and real harm. The size of that effect should depend on depth, not on how many parameters are trained.

What prior work establishes and leaves open

Several mechanistic studies motivate the claim that alignment is shallow. Safety Alignment Should be Made More Than Just a Few Tokens Deep measures the KL divergence between aligned and base models. It finds the difference concentrated in the first tokens and nearly gone by token 20, and shows that forcing a 10-token compliant opening (prefilling) raises Llama-2-7B attack success to 51.5%. Superficial Safety Alignment Hypothesis frames safety as a first-token "fulfill vs. refuse" classifier carried by roughly 1.3–1.4% of units. Robust LLM safeguarding via refusal feature adversarial training (ReFAT) shows that many jailbreaks work by suppressing a single "refusal feature", a direction in the residual stream, which is the running hidden state the model passes between layers. JULI finds that harmful continuations stay in the top-$k$ token probabilities even when the top choice is a refusal. Together these explain why a score read from the first token is such a good guide for an attacker.

Three works supply the method we reuse, and each also marks a limit. ReFAT replaces token-level adversarial search with ablation of the refusal direction during training. It reports roughly a 1,700× speedup over R2D2's online GCG loop (GCG being a gradient-based search for adversarial suffixes), but it updates model weights, assumes one refusal dimension, and does not evaluate query-based black-box attacks. Robustness Tokens trains about 10 vectors on a frozen vision transformer. It combines an invariance loss with an adversarial-correction loss, converges in about 200 steps, and keeps clean ImageNet accuracy at 85.2% against 85.6%. That paper is vision-only, and its robustness partly depends on the tokens staying secret, an assumption it never tests against repeated querying. Qi et al.'s safety-recovery data teaches mid-response pivots back to refusal from static synthetic examples, with no adversary in the loop.

Two further works are the closest neighbours, and they differ from ours in specific ways. STShield appends one sentinel token for detection and trains it against embedding-space noise. It detects attacks but does not change what the model generates, and its authors note it is untested against fluent semantic attacks. Safety Alignment Can Be Not Superficial With Explicit Safety Signals re-checks a safety classifier during decoding. It cannot be dropped onto a frozen model, because it requires changing the attention mask and retraining. Our proposal differs from all of these in three respects. It modifies generation behaviour while freezing every model weight. It trains against an adversary spread across response depth rather than at the first token. And it is evaluated mainly on whether it defeats query-based black-box optimization, including attackers who adapt to it.

Core mechanism

Let $\theta$ be the frozen model weights and $P \in \mathbb{R}^{k \times d}$ the $k$ trainable safety-token vectors of model width $d$. Let $A_r$ be an operator that projects an $r$-dimensional refusal subspace out of the residual stream at chosen layers. The subspace is estimated from differences in mean activation between harmful and harmless prompts, including paraphrased and non-English prompts, so that it is not limited to ReFAT's single English direction. Harmful training prompts $x$ are paired with a compliant prefix $c_{1:t}$ cut at a random depth $t \in \{0, \dots, T\}$ and a recovery target $y^{\text{ref}}$ that pivots to refusal. Benign prompts $x'$ come from instruction data plus XSTest-style "risky but benign" prompts. The objective is

$$\min_P \; \mathbb{E}_{x,t}\big[-\log p_\theta(y^{\text{ref}} \mid P, x, c_{1:t};\, A_r)\big] \;+\; \lambda\, \mathbb{E}_{x'}\big[\mathrm{KL}\big(p_\theta(\cdot \mid x') \,\|\, p_\theta(\cdot \mid P, x')\big)\big].$$

The first term asks the model, with its refusal directions removed and a harmful opening already written, to recover and refuse anyway at every depth. The second term, adapted from the Robustness Tokens invariance loss, keeps benign behaviour identical to the original model, with $\lambda$ weighting the two. The inner "attack" is a projection plus a teacher-forced prefix, both computed inside the forward pass, so each training step costs about the same as one ordinary fine-tuning step.

The mechanistic prediction follows from this setup. Score-based black-box attacks such as the random-search attack in JailbreakBench climb the log-probability of an affirmative first token. If the depth term works, a high first-token score will often be followed by recovery to refusal. The attacker's objective then decouples from harm judged on the full response, and query cost should rise even when first-token scores can still be pushed up.

Research Program

The work proceeds in four stages. First, we implement the method on Llama-2-7B-Chat, Gemma-7B and Vicuna-7B, each served behind a text-only interface that returns sampled text and, for score-based attacks, top-token log-probabilities. Second, we sweep the number of tokens $k \in \{1, 10, 32\}$, the subspace rank $r \in \{1, 4, 16\}$ and the maximum prefill depth $T \in \{0, 10, 40\}$. This separates the contributions of depth, subspace rank and token count. Third, we run a black-box attack suite with matched query budgets. Fourth, we analyse why the defense works or fails. That analysis covers the correlation between the attacker's proxy score and judge-rated harm, KL divergence by token position against the base model (repeating Qi et al.'s measurement), and whether successful attacks move activations along directions orthogonal to the trained subspace.

That last question is the open problem ReFAT leaves: does black-box search route around a low-dimensional refusal feature?

Baselines

The naive reference point is the undefended aligned model under the same attacks. JailbreakBench reports 90% attack success for "Prompt with RS" on Llama-2-7B, and we will reproduce that number before trusting our pipeline. A second fixed reference is an explicit safety system prompt with the same token budget as $P$, written in natural language rather than learned.

The strongest existing in-weights competitor is ReFAT. It is the closest method in cost and mechanism, and it reports HarmBench attack success under refusal-feature ablation on Gemma-7B falling from 92.8% to 18.8%. We found no reported result on Lacuna for ReFAT under query-based random search, PAIR or TAP, so we will measure those numbers ourselves. ReFAT will be run in two forms: full fine-tuning as published, and LoRA with an adapter whose parameter count is matched to ours. Both use the same harmful, benign and XSTest training data, the same number of optimizer steps, and an equal hyperparameter search budget of 20 trials per method, selected on a held-out validation split.

For deep alignment we run Qi et al.'s safety-recovery augmentation, which reports 10-token prefill attack success on Llama-2-7B dropping from 51.5% to 2.9%. We run it both in its published form and as soft tokens trained on the same recovery data without the ablation adversary. For inference-time defenses we include Gradient Cuff, which reports average attack success on Vicuna-7B-V1.5 falling from 76.7% to 25.7% across six attacks, while noting it requires embedding access. We also include STShield, which reports AmpleGCG attack success on Llama-2-7B-Chat dropping from 100% to 12%.

The ablations that isolate our key ingredient are depth-zero training ($T=0$, so ablation only, which amounts to ReFAT-on-tokens), rank one ($r=1$), and no adversary (recovery data only). Beating ReFAT means two things together. The first is lower HarmBench attack success at every query budget from 100 to 1,000 under adaptive random search and PAIR. The second is staying within one MMLU point and within the MT-Bench noise band of the undefended model, with an XSTest false-refusal rate no higher than ReFAT's. Both must hold while training cost stays within 2× of ReFAT's GPU-hours.

Why not just…?

Why not just run ReFAT with LoRA? This is the real threat. SSAH reports that LoRA damages safety-critical units during benign fine-tuning, but ReFAT-LoRA trained directly for safety may be fine. If ReFAT-LoRA matches us on black-box attack success and utility, then the token parameterization is not what matters. The contribution would reduce to the depth-and-subspace adversary, which we would then report as an add-on to ReFAT. The matched-parameter LoRA arm and the $T=0$ ablation settle this directly.

Why not just train on Qi et al.'s recovery data, without an adversary? Static recovery data already makes alignment deeper. If it alone breaks the proxy-to-harm link, the latent adversary is unnecessary. Our prediction is that recovery data without ablation fails once the attacker suppresses the refusal direction through fluent prompts, because nothing in training ever removed that direction. The no-adversary soft-token arm, evaluated under PAIR and TAP, tests exactly this.

Why not just filter inputs or outputs? Baseline Defenses for Adversarial Attacks Against Aligned Language Models shows that perplexity filtering and retokenization raise false rejections and reduce response quality. JailbreakBench notes that SmoothLLM fails against semantic attacks. Output judges add a second model call per response. Filters are complementary to our method rather than a substitute, and we report attack success with and without a HarmBench-classifier output filter stacked on each defense.

Why not just rely on the tokens being secret? Secrecy is part of what helped Robustness Tokens, and a white-box attacker who knows $P$ can optimize against it. We therefore claim only black-box robustness. We test whether that robustness comes from depth or from secrecy with a transfer attacker. This attacker trains its own copy of our method on a surrogate model, optimizes GCG against that copy, and then transfers the result to the deployed model.

Proposed Experiments

Harmful behaviours come from HarmBench's 510 behaviours and JBB-Behaviors, plus AdvBench for comparison with SSAH and STShield. Judging uses the HarmBench classifier, which reports 93.19% agreement with human labels. The attacks are prefilling at 5, 10 and 40 tokens, JailbreakBench random search with top-token log-probabilities, PAIR, TAP, AutoDAN, non-English rephrasings, and an adaptive random search that optimizes judge-scored harm on a sampled full response rather than first-token probability. The adaptive attack is our main stress test: if depth only hides the first-token signal, this attacker should recover. Utility is measured with MMLU, GSM8K, MT-Bench and AlpacaEval, and over-refusal with XSTest. Training cost is reported in GPU-hours and compared with ReFAT and with a short R2D2 run extrapolated to full length.

Three outcomes would decide the claim. The first is the proxy-decoupling test. The rank correlation between the attacker's first-token score and judged harm should fall sharply for the full method but not for the $T=0$ arm, and the attack-success-versus-query-budget curve should shift right accordingly. The second is the adaptive-attacker test. If full-response-scored random search restores attack success to the undefended level within 1,000 queries, the method only hides the signal rather than deepening refusal, and we would report a negative result. The third is the utility-and-cost test. Clean benchmarks must match the undefended model while training cost stays in ReFAT's range, because a gain bought with an alignment tax or an R2D2-scale budget would not answer the questions this note is asking.