Why Safety Alignment Fails Beyond the Initial Tokens

28 concepts16 papers1 proposal

When we interact with a safety-aligned language model, its refusal behavior feels like a robust conceptual boundary. If prompted to generate harmful content, the model typically responds with a standard boilerplate phrase: "I'm sorry, but I cannot fulfill this request."

However, a growing body of research reveals that this safety boundary is often incredibly thin—a phenomenon known as shallow safety alignment. Instead of deeply restructuring a model's internal representations to recognize and reject harmful concepts, standard alignment techniques primarily train the model to output specific refusal tokens at the very beginning of its response. Once an adversary or a generation sequence moves past this initial token window, the model's safety guardrails rapidly decay.

The Mechanics of Safety Decay

Why does safety alignment fade as a response progresses? The vulnerability stems from how autoregressive language models are trained and optimized.

During standard safety alignment (such as RLHF or DPO), the optimization gradient heavily concentrates on the transition from the prompt to the refusal prefix. As a result, the model learns a superficial shortcut: it associates safety with starting a response correctly, rather than evaluating the ongoing generation. As demonstrated by Xiangyu Qi et al. in Safety Alignment Should be Made More Than Just a Few Tokens Deep and corroborated by Wonje Jeung et al. in A2D: Any-Order, Any-Step Safety Alignment for Diffusion Language Models, the token-level probability distribution of an aligned model quickly converges back to that of its unaligned base version after the first few tokens.

[Prompt] -> [Initial Tokens: "I am sorry..."] -> [Safety Decay] -> [Base Model Distribution Takeover]

Thong Bach et al. in Rethinking Deep Alignment Through The Lens Of Incomplete Learning trace this phenomenon to gradient signal decay during autoregressive training, which leaves the structural tail of longer sequences severely undertrained. Furthermore, this superficial alignment is heavily anchored to structural artifacts rather than semantic understanding. Chak Tou Leong et al. discovered in Why Safeguarded Ships Run Aground? Aligned Large Language Models' Safety Mechanisms Tend to Be Anchored in The Template Region that safety mechanisms rely on fixed chat templates (the boilerplate formatting separating user prompts and model responses) to aggregate harm signals. If an adversarial prompt disrupts these structural anchors, the internal safety logic collapses.

Crucially, optimizing purely for token probabilities creates a facade of safety. As argued by Jason Vega et al. in Matching Ranks Over Probability Yields Truly Deep Safety Alignment, standard training forces a refusal token to have the highest absolute probability without fundamentally shifting the underlying ranking of dangerous alternative tokens. The latent capability to generate harm remains fully intact just beneath the surface.

Exploiting the Structural Tail

Because safety is concentrated at the start of a response, disrupting these initial tokens completely disarms the model's guardrails.

The most direct exploitation of this vulnerability is the prefilling attack, where an attacker forces the model to begin its response with an affirmative phrase like "Sure, here is...". Xuandong Zhao et al. in Weak-to-Strong Jailbreaking on Large Language Models and Jiayou Wang et al. in SecDecoding: Steerable Decoding for Safer LLM Generation show that once the decoding process is steered past the initial refusal phase, the internal probability distributions for safe and unsafe outputs become nearly indistinguishable, allowing the model to complete harmful requests unimpeded.

This token-sequence memorization is so fragile that it can be broken using entirely benign data. Yangyang Guo et al. demonstrated in LLMs Can Unlearn Refusal with Only 1,000 Benign Samples that fine-tuning a model on just 1,000 safe answers prepended with refusal prefixes (e.g., "I'm sorry, here is the recipe for apple pie") causes the model to unlearn the association between the refusal trigger and actual rejection behavior.

Adversaries can also exploit the structural tail of a generation without forcing an affirmative prefix. Joshua Kazdan et al. in No, of Course I Can! Deeper Fine-Tuning Attacks That Bypass Token-Level Safety Mechanisms introduced a "refuse-then-comply" attack strategy, showing that production models can be manipulated to generate a safe-looking refusal prefix but still deliver harmful content later in the same sequence.

This structural mismatch also creates hurdles for model editing. The authors of DualEdit: Mitigating Safety Fallback in LLM Backdoor Editing via Affirmation-Refusal Regulation observed a "safety fallback" phenomenon where backdoored models begin executing a harmful command but abruptly revert to a refusal mid-sentence, because the backdoor injection technique only optimized the likelihood of the initial tokens.

Refuse-then-Comply Attack: [Prompt] -> ["I cannot help with that..."] -> [Safety Decays] -> [Harmful Content]
Safety Fallback Behavior:  [Prompt] -> [Harmful Content Starts] -> [Original Safety Regains Control] -> ["Refusal"]

The Fragility of Downstream Fine-Tuning

Because shallow safety alignment behaves like a thin veneer, it is exceptionally vulnerable to being washed away during downstream fine-tuning—even on completely benign datasets.

When a model is fine-tuned for specific capabilities, the training data often alters the expected structure of the initial tokens. For example, Dongyoon Hahm et al. in Unintended Misalignment from Agentic Fine-Tuning: Risks and Mitigation found that fine-tuning models on agentic tasks (like web navigation or coding) drastically erodes safety guardrails. Because agentic datasets train the model to immediately output a tool call or action sequence rather than a text prefix, the critical decision point for refusal is bypassed entirely.

Similarly, Aladin Djuhera et al. demonstrated in SafeCOMM: A Study on Safety Degradation in Fine-Tuned Telecom Large Language Models that specialized domain fine-tuning restricts the safety influence to the opening tokens, leaving the rest of the generation trajectory highly vulnerable to domain-specific exploitation.

Parallel and Non-Autoregressive Vulnerabilities

Shallow alignment is not merely an artifact of left-to-right autoregressive generation; it also manifests in non-linear generative architectures like Diffusion Large Language Models (dLLMs).

In iterative diffusion generators, safety alignment is heavily front-loaded into the first few denoising steps. Arth Singh demonstrated in Re-Mask and Redirect: Exploiting Denoising Irreversibility in Diffusion Language Models that if an attacker hijacks or overwrites these initial sampling steps—deleting early refusal text and replacing it with an affirmative prefix—the model's subsequent denoising steps are easily steered toward harmful outputs.

Furthermore, because dLLMs decode tokens in parallel, their safety training focuses primarily on refusing to initiate a harmful response rather than refusing to complete one. As a result, Zichen Wen et al. showed in The Devil behind the mask: An emergent safety vulnerability of Diffusion LLMs that diffusion models fail to maintain safety when tasked with infilling masked spans within a pre-structured harmful template, bypassing the model's front-loaded refusal path entirely.

Moving Toward Deep Alignment

To move past superficial token-sequence memorization, recent work focuses on formalizing and enforcing safety depth across the entire generation trajectory.

Ching-Chia Kao et al. in Safety Depth in Large Language Models: A Markov Chain Perspective formalize safety depth using Markov Chain theory, mapping out the exact token positions where a model reliably maintains or loses its refusal state. To prevent the model from drifting into harmful states during long or multi-turn conversations, they introduce cyclic data augmentation strategies that train the model to maintain safety boundaries at any point in the sequence.

Other approaches intervene directly during inference or optimization. For instance, the token-level defense proposed by Wonje Jeung et al. trains models to emit an immediate end-of-sentence signal the moment harmful content appears anywhere in the generation trajectory, regardless of when or where it occurs. By shifting safety alignment from a front-loaded prefix toward continuous trajectory regulation, these methods aim to ensure that safety remains a deep property of the model's reasoning rather than a temporary illusion.

Key Foundational Work

A2D: Any-Order, Any-Step Safety Alignment for Diffusion Language Models (2026) — contributed 4 concept elements to this direction.
No, of Course I Can! Deeper Fine-Tuning Attacks That Bypass Token-Level Safety Mechanisms (2026) — contributed 2 concept elements to this direction.
Safety Alignment Should be Made More Than Just a Few Tokens Deep (2025) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_shallow_safety_alignment_8097687c,
  title = {shallow safety alignment},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/shallow-safety-alignment/rdr_8097687cabdf4829881b022248e1bcee},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_shallow_safety_alignment_8097687c_v1,
  title = {shallow safety alignment},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/shallow-safety-alignment/rdr_8097687cabdf4829881b022248e1bcee/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake