Safety Alignment Depth in Large Language Models: A Markov Chain Perspective

Ching-Chia Kao, Chia-Mu Yu, Chun-Shien Lu, Chu-Song ChenarXiv 2025 · DOI · arXiv

Introduction

As Large Language Models (LLMs) become increasingly integrated into critical infrastructure, the robustness of their safety mechanisms is under intense scrutiny. Current alignment techniques, such as Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO), often result in "shallow" safety. While these models may initially refuse a harmful prompt (e.g., starting a response with "I cannot fulfill this request"), they frequently revert to unsafe behavior if an attacker can bypass these initial tokens. This paper introduces a rigorous mathematical framework to address this vulnerability, defining the concept of "Safety Depth" through the lens of Markov Chains.

Theoretical Framework: LLMs as Markov Chains

The authors bridge the gap between discrete probability and deep learning by modeling the autoregressive generation process of an LLM as a Markov Chain. In this view, each token generation is a transition between states in a sequence.

$\delta$-absorbing States

To quantify safety, the researchers define "safe" states as $\delta$-absorbing. Mathematically, if a state (or sequence of tokens) is $\delta$-absorbing, the probability of the model transitioning from a refusal state back into a harmful output state is less than a negligible threshold $\delta$. A model with high Safety Depth is one where refusal states act as inescapable "traps," ensuring that once a model begins to refuse, it remains in a safe state throughout the remainder of the generation.

The Bias Matrix

Fine-tuning for safety is conceptualized as adding a "bias matrix" to the pre-trained Markov transition matrix. This allows the authors to calculate the exact number of training steps ($T$) required to transform a standard state into an unbreakable refusal state.

Proposed Methodologies

The paper proposes two primary strategies to deepen safety alignment: Cyclic Group Augmentation and Ensemble Theory.

Cyclic Group Augmentation

Traditional safety training often relies on static refusal phrases. The authors argue this is insufficient for creating depth. Instead, they employ group theory to create "cyclic" permutations of refusal data. By rotating phrases—for example, shifting the sequence "{I, cannot, fulfill, this}" across different token positions—the model learns a generalized "refusal manifold." This ensures the model maintains its safety posture regardless of where the refusal logic is triggered in the sequence.

Ensemble Robustness

The researchers provide the first theoretical proof demonstrating how ensemble "width" (the number of models) can compensate for individual "depth." They analyze three specific strategies:

  1. Union: Safety is maintained if any model in the ensemble refuses.
  2. Average: Safety is determined by the mean probability across the ensemble.
  3. Majority Voting: The most robust method, where the collective decision of the group satisfies a safety threshold $\epsilon$ even if individual models are marginally weaker.

Experimental Results

The framework was tested using open-source models including Gemma 2B, Phi-2, and Qwen 2.5. Training was conducted on the MaliciousInstruct dataset, while evaluation utilized the HEx-PHI dataset, covering 11 prohibited categories such as malware and physical harm.

Key Findings

  • Superiority of Cyclic Alignment: In all tested models, the Cyclic Group Augmentation strategy significantly outperformed both "shallow" and "deep" (standard repetitive) alignment. For instance, on the Gemma 2B model, the mean safety score improved from 0.35 (unaligned) to 0.61 (cyclic).
  • Ensemble Performance: Combining models proved highly effective. Using majority voting, safety scores clustered between 0.9 and 1.0, effectively eliminating the inconsistent safety responses often seen in single-model deployments.
  • Exponential Decay of Risk: Experimental data confirmed the theoretical prediction that "escape probabilities" (the chance of a model leaving a refusal state) decay exponentially during the training process.

Limitations and Future Directions

While the mathematical foundations are robust, the authors acknowledge certain limitations. The experiments were primarily conducted on smaller models (1.5B to 2B parameters); the training dynamics of 70B+ parameter models may present additional complexities. Furthermore, there is an inherent "utility trade-off" where aggressive safety depth could lead to "moral over-refusal" of benign prompts. Future research will focus on balancing this newfound safety depth with the model's helpfulness and general utility in real-world applications.

Conclusion

By shifting the focus from behavioral observation to mathematical guarantees, this research provides a path toward provably safe LLMs. The transition from "shallow" alignment to "Safety Depth" via Markov Chain modeling offers a principled way to defend against adversarial jailbreaks and ensure models remain aligned throughout the entire generation process.

Summary updated

Abstract

Large Language Models (LLMs) are increasingly adopted in high-stakes scenarios, yet their safety mechanisms often remain fragile. Simple jailbreak prompts or even benign fine-tuning can bypass these protocols, underscoring the need to understand where and how they fail. Recent findings suggest that vulnerabilities emerge when alignment is confined to only the initial output tokens. Unfortunately, even with the introduction of deep safety alignment, determining the optimal safety depth remains an unresolved challenge. By leveraging the equivalence between autoregressive language models and Markov chains, this paper offers the first theoretical result on how to identify the ideal depth for safety alignment, and demonstrates how permutation-based data augmentation can tighten these bounds. Crucially, we reveal a fundamental interaction between alignment depth and ensemble width-indicating that broader ensembles can compensate for shallower alignments. These insights provide a theoretical foundation for designing more robust, scalable safety strategies that complement existing alignment approaches, opening new avenues for research into safer, more reliable LLMs.
BibTeX
@misc{kao2025safety,
  author = {Ching-Chia Kao and Chia-Mu Yu and Chun-Shien Lu and Chu-Song Chen},
  title = {{Safety Alignment Depth in Large Language Models: A Markov Chain Perspective}},
  year = {2025},
  eprint = {2502.00669},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2502.00669},
}
Figure