Token Level Safety Steering during Language Model Decoding

24 concepts20 papers4 proposals

Traditional safety alignment for Large Language Models (LLMs) operates under a static paradigm: models are fine-tuned via reinforcement learning or supervised instruction to reject harmful prompts, or external filters screen inputs and outputs. However, this approach creates a fragile shell of "shallow safety."

If an adversarial prompt manages to crack this outer shell, the model's internal safety guardrails often collapse entirely. To address this vulnerability, a new paradigm has emerged: token-level inference-time safety steering. Instead of treating safety as a binary pre- or post-generation check, this approach dynamically monitors and corrects the model's internal states and token probabilities at every single step of the autoregressive decoding process.

The Failure of Static Alignment

To understand why token-level steering is necessary, we must look at how autoregressive generation betrays traditional safety training.

As identified by Yuan et al., standard safety alignment suffers from a severe refusal position bias. Because models are typically trained to refuse harmful prompts at the very beginning of a response (e.g., "I cannot fulfill this request"), they struggle to pivot to a safe stance if they have already begun generating text.

Adversarial attacks exploit this exact blind spot. For instance, Wang et al. demonstrate that forcing an LLM to start its response with a positive prefix (such as "Sure, here is...") completely bypasses initial safety filters. Once the model emits or accepts these initial tokens, it becomes trapped by the statistical gravity of its own context window. It prioritizes logical coherence with the unsafe prefix over its global safety training, doubling down on generating harmful content.

[Adversarial Prompt] ──> [Input Filter: Passed] ──> [Forced Prefix: "Sure!"] ──> [Statistical Gravity] ──> [Unsafe Output]

Furthermore, malicious intent is not always obvious at the prompt level. Cui et al. show that entirely benign inputs can combine to trigger an unsafe "safety snowball effect" mid-generation as the model's reasoning chain unfolds. As Li and Kim argue in their Superficial Safety Alignment Hypothesis, static frameworks fail to monitor these dynamic shifts, leaving long-form generations highly vulnerable to safety drifts.

Continuous Token-Level Evaluation

Token-level safety steering shifts the defense strategy from absolute prevention to real-time verification during decoding, as surveyed by Dong et al.. Rather than evaluating a sequence as a single static unit, the system treats safety as an evolving trajectory (Peng et al.).

During autoregressive decoding, the model computes a probability distribution (logits) over the vocabulary to select the next token. Token-level steering inserts an evaluation loop directly into this step.

[Current Context] ──> [Compute Logits] ──> [Evaluate Safety Trajectory] ──> [Steer/Modify Logits] ──> [Sample Safe Token]

This evaluation can happen through multiple lenses:

  • Internal State Probing: Zeng et al. leverage the model's own hidden activations to inform a classifier that assesses the harmfulness of candidate tokens before they are finalized.
  • Process Reward Models (PRMs): Instead of scoring the final output, a specialized reward model evaluates multiple candidate continuations at each individual step to filter out insecure implementation paths in real-time (Yu et al.).
  • Deep Integration: By embedding safety checks directly into the inference process (Cunningham et al.), systems can detect the precise token where a generation transitions from benign to harmful.

Mechanisms of Intervention

Once an unsafe trajectory or token candidate is detected mid-generation, the framework can intervene using three primary strategies: controlled decoding, mid-sequence halting, or backtracking.

1. Controlled Decoding and Logit Modification

Instead of stopping the model, the system can gently steer it. Zeng et al. and Ghosal et al. use a safety reward model to dynamically adjust token logits during sampling. If a candidate token leans toward a harmful path, its logit score is suppressed, automatically elevating safer alternatives or guiding the model toward a polite disclaimer without altering its underlying weights.

2. Mid-Sequence Halting and Signal Injection

If a generation crosses a risk threshold, the system can force an immediate pivot. Li and Kim utilize explicit safety signals—such as a dedicated token—to continuously monitor decoding and inject a refusal phrase the moment a violation occurs. Similarly, the Any-Depth Alignment (ADA) framework proposed by Zhang et al. injects safety tokens mid-generation to probe hidden states, allowing the system to halt harmful outputs at any depth with minimal computational overhead.

3. Backtracking and Rewinding

Because discrete evaluation intervals can sometimes let toxic tokens slip through momentarily, some frameworks introduce a "rewind" mechanism. Zhang et al. break the one-way nature of autoregressive generation by allowing the model to perform a corrective reset when an unsafe prefix begins to emerge.

Taking this further, Sel et al. introduce Reinforcement Learning with Backtracking Feedback (RLBF), which trains models to recognize their own mid-generation violations, issue an explicit "backtrack" command, and seamlessly rewrite the problematic segment. This self-monitoring loop is mirrored in the Cognition-of-Thought (CooT) framework by Zhang et al., which dynamically rewinds reasoning paths to align with safety priorities.

[Safe Text] ──> [Unsafe Token Detected!] ──> [Trigger Backtrack] ──> [Erase Unsafe Tokens] ──> [Rewrite Safe Path]

Why It Works: The Intuition of Real-Time Correction

The core reason token-level steering succeeds where input filtering fails lies in the difference between predicting harm and observing harm.

Complex jailbreaks are designed to mask malicious intent behind multi-layered, abstract contexts that appear entirely benign to an input filter. However, for the LLM to actually fulfill the malicious request, it must eventually translate that abstract context into concrete, harmful tokens.

The moment the model begins this translation, its internal hidden states change, and the token probabilities shift. By monitoring the generation process at the token level, safety mechanisms leverage the model's own unfolding reasoning chains against the attack. It decouples safety from the initial prompt state, ensuring that no matter how a model is tricked into starting an unsafe response, it retains the ability to recognize the danger and steer itself back to safety.

Key Foundational Work

SAFENUDGE: Safeguarding Large Language Models in Real-time with Tunable Safety-Performance Trade-offs (2025) — contributed 3 concept elements to this direction.
ROOT DEFENCE STRATEGIES: ENSURING SAFETY OF LLM AT THE DECODER LEVEL (2025) — contributed 3 concept elements to this direction.
Safety Alignment Can Be Not Superficial With Explicit Safety Signals (2025) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_inference_time_safety_steering_b989a5d6,
  title = {inference-time safety steering},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/inference-time-safety-steering/rdr_b989a5d6e7d0432b9bba001dd922d38d},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_inference_time_safety_steering_b989a5d6_v1,
  title = {inference-time safety steering},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/inference-time-safety-steering/rdr_b989a5d6e7d0432b9bba001dd922d38d/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake