Abstract
Current safety alignment techniques for Large Language Models (LLMs), such as Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO), often result in "shallow" safety guardrails. These guardrails primarily influence the initial tokens of a response, creating a vulnerability where models can be forced past these refusal tokens to produce harmful content. This paper analyzes this phenomenon through per-token KL divergence and gradient norms, revealing that safety signals are heavily concentrated at the start of sequences. We propose "deepening" alignment through safety-recovery data augmentation and a token-wise constrained fine-tuning objective, significantly improving robustness to jailbreaking and fine-tuning attacks while preserving general utility.
Introduction
Despite rigorous safety training, LLMs remain susceptible to adversarial attacks, including adversarial suffixes (GCG), response prefilling, and low-resource fine-tuning on benign data. Researchers from Princeton University and Google DeepMind argue that this is because current alignment methods act as a "shortcut." Instead of fundamentally altering the model's internal representation of harmful content, these methods primarily shift the probability distribution of the first few output tokens. If an attacker can bypass these initial "refusal" tokens, the rest of the model’s generative engine remains largely unaligned and capable of generating harmful outputs.
The Shallow Alignment Phenomenon
To investigate this "shallow alignment," the researchers performed a per-token analysis of the KL divergence between aligned models (e.g., Llama-2-7B-Chat) and their unaligned base versions. The results, shown in Figure 1, demonstrate that the "safety signal" is extremely high for the first 5–10 tokens but drops precipitously thereafter. Beyond this initial window, the aligned model's distribution becomes remarkably similar to the unaligned base model.
This suggests that the safety observed in chat models is often a pattern-following behavior triggered by the model's own initial refusal tokens. If the model is forced to start a response with harmful content, the safety guardrails effectively collapse.
Vulnerability to Prefilling Attacks
The impact of this shallow alignment is clearly visible in prefilling attacks. By forcing the model to begin its response with a specific number of harmful tokens, attackers can bypass the initial refusal mechanism. As shown in Figure 2, the Attack Success Rate (ASR) increases significantly as the number of prefilled harmful tokens grows, even for models that have undergone standard safety alignment.
Proposed Mitigations: Deepening Safety
The authors propose two primary methods to create more robust, "deep" safety alignment:
- Safety Recovery Augmentation: This involves training the model on synthetic data where a response begins with harmful content but then "recovers" into a refusal (e.g.,
[Harmful Step 1] I cannot fulfill your request...). This teaches the model to return to a safe state even if it has already begun a harmful trajectory. - Token-wise Constrained Fine-tuning: This new optimization objective applies a per-token regularization parameter ($\beta_t$). By applying high regularization to the first few tokens, the model's refusal behavior is "locked" during downstream fine-tuning, preventing the model from "unlearning" its safety guardrails when exposed to new data.
Key Results and Utility Preservation
The proposed "Augmented" Llama-2-7B-Chat model demonstrated significantly higher robustness. In tests involving 40 prefilled harmful tokens, the standard model reached a 57% ASR, while the augmented model remained at 4.5%. Furthermore, the constrained fine-tuning objective successfully defended against fine-tuning attacks; while a standard attack on Llama-2-7B-Chat reached 88.9% ASR, the constrained version stayed at 4.6%.
Crucially, these safety improvements did not degrade the models' general capabilities. Performance on utility benchmarks, such as GSM8k (math) and Samsum (summarization), remained comparable to standard aligned models, proving that safety can be deepened without sacrificing intelligence.
Conclusion and Future Work
This research highlights a critical flaw in current LLM alignment: its reliance on a shallow, token-position-dependent shortcut. By implementing recovery training and token-wise constraints, developers can build models that are fundamentally more resistant to adversarial manipulation. Future work should explore integrating these deep objectives directly into the initial RLHF and DPO training phases to ensure safety is baked into the model's core generative process.
Summary updated