Test-Time Safety Alignment

Baturay Saglam, Dionysis KalogeriasarXiv 2026 · arXiv

TL;DR

Test-Time Safety Alignment (TSA) neutralizes jailbreaks and harmful prompts by optimizing an LLM's input embeddings at inference time using zeroth-order gradients from a black-box moderation API. Remarkably, these continuous adjustments are "sub-lexical"—they steer the model to generate safe responses without changing the human-readable text of the prompt or introducing a blanket refusal bias.

What Problem Does This Paper Address?

Large language models undergo extensive safety alignment (such as RLHF or DPO) during training, yet adversarial "jailbreak" prompts can still exploit gaps in this alignment to elicit harmful content.

Existing test-time defenses typically manipulate discrete text tokens (which can be brittle and easily bypassed), require training separate auxiliary guard models, or apply static internal activation steering. This paper addresses the lack of a direct, continuous optimization method at the input level that can dynamically adapt to any prompt at test time without altering the underlying model's weights.

Methods

The authors introduce Test-Time Safety Alignment (TSA). Instead of modifying the text tokens of a prompt, TSA optimizes the continuous prompt embedding matrix that the model actually conditions on.

Because both the target LLM and the safety evaluator (the OpenAI Moderation API) are treated as black boxes, standard backpropagation is impossible. To get around this, TSA uses a zeroth-order gradient estimator:

  1. It applies random Gaussian noise to the prompt embeddings to create multiple perturbed copies.
  2. It runs forward passes through the LLM and queries the moderation API to see how the safety scores change.
  3. It uses these differences to estimate a gradient direction and updates the embeddings via gradient descent.

To prevent the embeddings from drifting into regions of latent space the model hasn't seen during training, TSA incorporates two simple regularization mechanisms: gradient normalization (ensuring consistent step sizes) and a cosine similarity constraint (projecting the embeddings back if they stray too far from the original prompt).

Datasets

The method was evaluated on two prominent red-teaming benchmarks:

  • WildJailbreak: The authors utilized the adversarial harmful split (2,000 prompts rewritten with stealthy, in-the-wild jailbreak tactics) and the adversarial benign split (210 prompts that mimic jailbreak structures but have harmless intent, serving as a control).
  • HarmBench: A dataset of 400 direct, explicit harmful queries violating standard AI safety policies.

Key Insights and Ideas

  • Sub-Lexical Control: Surprisingly, the optimized embeddings still decode back to the exact same original text tokens. The perturbation is small enough to stay within each token's Voronoi cell, meaning the human-readable prompt remains identical, yet the subtle continuous shift completely redirects the LLM's internal hidden states toward safe generation.
  • Asymmetry of Safety Landscapes: The authors attempted to reverse the optimization to generate jailbreaks, but it failed. Aligned models sit within a broad "refusal basin" where local perturbations continue to trigger refusals. The narrow compliance pathways exploited by sophisticated jailbreaks lie beyond the reach of local isotropic probes.
  • No Blanket Refusal Bias: TSA employs early stopping (halting optimization once the moderation score drops below 0.1). Because it only intervenes when a prompt actually triggers a safety risk, it leaves benign prompts entirely unaffected, avoiding the over-refusal tendencies of other defenses.

Key Results

  • Complete Neutralization: TSA completely eliminated safety-flagged responses (reducing the flagged count to 0) for 4 out of 5 tested models (GPT-OSS-20B, Qwen3-14B, Llama 3.1-8B, and Phi-3.5-4B) across both HarmBench and WildJailbreak.
  • Rapid Convergence: For well-aligned models like Llama 3.1-8B, the optimization typically converged in a single gradient step.
  • Gemma 3 Exception: Gemma 3-1B was the sole exception, failing to resolve a fraction of harmful prompts. The authors hypothesize this is due to its smaller embedding dimension (1152) or thinner baseline safety training.
  • Preserving Benign Inputs: On the adversarial benign control split, already-safe responses were left substantively intact, outperforming baseline defenses like SmoothLLM, AdaSteer, and RESTA, which occasionally caused false positives or increased moderation scores.

The complete breakdown of these results across models and baselines can be seen in Table 1:

Table 1: Text moderation results for five instruction-tuned language models on two red-teaming benchmarks. Responses are generated at a sampling temperature of 0.1 with three independent trials per prompt. Scores report statistics of the maximum over the 13 OpenAI Moderation API harm categories per prompt, in [0, 1]; lower is safer. Optimization runs for up to 10 iterations or until the maximum score across the 13 categories falls below 0.1. Flagged counts the prompts whose completions the API classifies as harmful. Time reports the average per-prompt optimization wall-clock duration, and Net excludes moderation API latency. Within each (model, benchmark) pair, the best value for each metric is shown in bold.

Limitations and Future Work

  • Per-Prompt Latency: The optimization loop introduces an overhead of several seconds per prompt. This is primarily driven by network round-trips to the cloud-based OpenAI Moderation API. Utilizing a local moderation classifier could drastically reduce this latency.
  • Embedding Access Requirement: TSA requires access to the input word embedding matrix. While this condition is easily met for open-weight models, it cannot be applied to purely API-gated models (like GPT-4 or Claude) where users can only pass discrete text.
  • Dependence on Base Alignment: TSA amplifies existing safety alignment rather than replacing it entirely; if a model's baseline safety training is too thin (as seen with Gemma 3), the optimizer struggles to find a safe direction within the cosine ball.
  • Future Directions: A natural next step is exploring whether sub-lexical embedding optimization can be generalized to steer other test-time behaviors, such as strict instruction-following or complex policy compliance.

Summary updated

Abstract

Recent work has shown that a model's input word embeddings can serve as effective control variables for steering its behavior toward outputs that satisfy desired properties. However, this has only been demonstrated for pretrained text-completion models on the relatively simple objective of reducing surface-level profanity in short continuations. A natural and practically important question is how well input embeddings can control aligned models, which produce an imbalanced bimodal refuse-or-comply output distribution rather than the smooth distribution characteristic of open-ended generation. We explore this in the context of safety, showing that input word embeddings can be optimized in a sub-lexical manner to minimize the semantic harmfulness of aligned model responses. Our approach uses zeroth-order gradient estimation of a black-box text-moderation API with respect to the input embeddings, and then applies gradient descent on these embeddings to minimize the harmfulness of the generated text. Experiments show that the proposed method can neutralize every safety-flagged response on standard safety benchmarks.
BibTeX
@misc{saglam2026test,
  author = {Baturay Saglam and Dionysis Kalogerias},
  title = {{Test-Time Safety Alignment}},
  year = {2026},
  eprint = {2604.26167},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2604.26167},
}

Loading related content...

Figure