Safety Depth in Large Language Models: A Markov Chain Perspective

Ching-Chia Kao, Chia-Mu Yu, Chun-Shien Lu, Chu-Song ChenNeurIPS 2025 · OpenReview

Introduction

As Large Language Models (LLMs) become increasingly integrated into critical infrastructure, the robustness of their safety mechanisms is under intense scrutiny. Current alignment techniques, such as Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO), often result in "shallow" safety. While these models may initially refuse a harmful prompt (e.g., starting a response with "I cannot fulfill this request"), they frequently revert to unsafe behavior if an attacker can bypass these initial tokens. This paper introduces a rigorous mathematical framework to address this vulnerability, defining the concept of "Safety Depth" through the lens of Markov Chains.

Theoretical Framework: LLMs as Markov Chains

The authors bridge the gap between discrete probability and deep learning by modeling the autoregressive generation process of an LLM as a Markov Chain. In this view, each token generation is a transition between states in a sequence.

$\delta$-absorbing States

To quantify safety, the researchers define "safe" states as $\delta$-absorbing. Mathematically, if a state (or sequence of tokens) is $\delta$-absorbing, the probability of the model transitioning from a refusal state back into a harmful output state is less than a negligible threshold $\delta$. A model with high Safety Depth is one where refusal states act as inescapable "traps," ensuring that once a model begins to refuse, it remains in a safe state throughout the remainder of the generation.

The Bias Matrix

Fine-tuning for safety is conceptualized as adding a "bias matrix" to the pre-trained Markov transition matrix. This allows the authors to calculate the exact number of training steps ($T$) required to transform a standard state into an unbreakable refusal state.

Proposed Methodologies

The paper proposes two primary strategies to deepen safety alignment: Cyclic Group Augmentation and Ensemble Theory.

Cyclic Group Augmentation

Traditional safety training often relies on static refusal phrases. The authors argue this is insufficient for creating depth. Instead, they employ group theory to create "cyclic" permutations of refusal data. By rotating phrases—for example, shifting the sequence "{I, cannot, fulfill, this}" across different token positions—the model learns a generalized "refusal manifold." This ensures the model maintains its safety posture regardless of where the refusal logic is triggered in the sequence.

Ensemble Robustness

The researchers provide the first theoretical proof demonstrating how ensemble "width" (the number of models) can compensate for individual "depth." They analyze three specific strategies:

  1. Union: Safety is maintained if any model in the ensemble refuses.
  2. Average: Safety is determined by the mean probability across the ensemble.
  3. Majority Voting: The most robust method, where the collective decision of the group satisfies a safety threshold $\epsilon$ even if individual models are marginally weaker.

Experimental Results

The framework was tested using open-source models including Gemma 2B, Phi-2, and Qwen 2.5. Training was conducted on the MaliciousInstruct dataset, while evaluation utilized the HEx-PHI dataset, covering 11 prohibited categories such as malware and physical harm.

Key Findings

  • Superiority of Cyclic Alignment: In all tested models, the Cyclic Group Augmentation strategy significantly outperformed both "shallow" and "deep" (standard repetitive) alignment. For instance, on the Gemma 2B model, the mean safety score improved from 0.35 (unaligned) to 0.61 (cyclic).
  • Ensemble Performance: Combining models proved highly effective. Using majority voting, safety scores clustered between 0.9 and 1.0, effectively eliminating the inconsistent safety responses often seen in single-model deployments.
  • Exponential Decay of Risk: Experimental data confirmed the theoretical prediction that "escape probabilities" (the chance of a model leaving a refusal state) decay exponentially during the training process.

Limitations and Future Directions

While the mathematical foundations are robust, the authors acknowledge certain limitations. The experiments were primarily conducted on smaller models (1.5B to 2B parameters); the training dynamics of 70B+ parameter models may present additional complexities. Furthermore, there is an inherent "utility trade-off" where aggressive safety depth could lead to "moral over-refusal" of benign prompts. Future research will focus on balancing this newfound safety depth with the model's helpfulness and general utility in real-world applications.

Conclusion

By shifting the focus from behavioral observation to mathematical guarantees, this research provides a path toward provably safe LLMs. The transition from "shallow" alignment to "Safety Depth" via Markov Chain modeling offers a principled way to defend against adversarial jailbreaks and ensure models remain aligned throughout the entire generation process.

Summary updated

Abstract

Large Language Models (LLMs) are increasingly adopted in high-stakes scenarios, yet their safety mechanisms often remain fragile. Simple jailbreak prompts or even benign fine-tuning can bypass internal safeguards, underscoring the need to understand the failure modes of current safety strategies. Recent findings suggest that vulnerabilities emerge when alignment is confined to only the initial output tokens. To address this, we introduce the notion of safety depth, a designated output position where the model refuses to generate harmful content. While deeper alignment appears promising, identifying the optimal safety depth remains an open and underexplored challenge. We leverage the equivalence between autoregressive language models and Markov chains to derive the first theoretical result on identifying the optimal safety depth. To reach this safety depth effectively, we propose a cyclic group augmentation strategy that improves safety scores across six LLMs. In addition, we uncover a critical interaction between safety depth and ensemble width, demonstrating that larger ensembles can offset shallower alignments. These results suggest that test-time computation, often overlooked in safety alignment, can play a key role. Our approach provides actionable insights for building safer LLMs.
BibTeX
@inproceedings{kao2025safety,
  author = {Ching-Chia Kao and Chia-Mu Yu and Chun-Shien Lu and Chu-Song Chen},
  title = {{Safety Depth in Large Language Models: A Markov Chain Perspective}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=tu3P6KSHGN},
}
Figure