Attributing and Exploiting Safety Vectors through Global Optimization in Large Language Models

Fengheng Chu, Jiahao Chen, Yuhong Wang, Jun Wang, Zhihui Fu, Shouling Ji, Songze LiarXiv 2026 · arXiv

TL;DR

The paper introduces GOSV (Global Optimization for Safety Vector Extraction), a framework that uncovers how LLM safety mechanisms are distributed across attention heads. By optimizing over all heads simultaneously, the authors discover two distinct internal safety pathways and reveal that disrupting roughly 30% of a model's attention heads completely dismantles its safety guardrails, enabling highly effective white-box jailbreaks without any prompt engineering.

What Problem Does This Paper Address?

Despite safety alignment techniques like RLHF and DPO, Large Language Models (LLMs) remain notoriously vulnerable to jailbreak attacks. To build better defenses, we need a mechanistic understanding of where and how safety is processed inside the model.

Prior interpretability work typically relies on "local, greedy attribution"—evaluating components (like individual attention heads) one by one in isolation. However, LLMs are highly recurrent, interdependent networks where information flows dynamically. Local methods fail to capture emergent safety behaviors that arise from cooperative interactions between multiple attention heads. This paper addresses this gap by treating safety attribution as a global optimization problem.

Methods

The authors propose GOSV, a framework that identifies safety-critical attention heads by optimizing over the entire space of attention heads simultaneously using a REINFORCE-based reinforcement learning algorithm.

Figure 1: Overview of our approach: GOSV framework and our attack. Left: The GOSV optimization process employs two activation repatching strategies (Harmful Patching and Zero Ablation) combined with global optimiza- tion to identify safety-critical attention heads. Right: Our attack replaces activations at the identified safety-critical heads with the corresponding repatching values at inference time.

GOSV employs two complementary activation repatching strategies at the last token position during the forward pass:

  1. Harmful Patching: Injects the mean activation values calculated from a dataset of harmful generations. This introduces statistical signatures that trigger unsafe outputs, resulting in Malicious Injection Vectors.
  2. Zero Ablation: Sets the activations of selected heads to zero, effectively killing the information flow and eliminating safety-relevant signals, resulting in Safety Suppression Vectors.

Instead of manually tweaking prompts, GOSV learns a Bernoulli distribution over all attention head locations, optimizing the selection to minimize the semantic distance (measured via embedding cosine similarity) between the model's output and a target harmful response.

Datasets

The framework was trained and evaluated using two prominent safety benchmarks:

  • AdvBench: The authors used a tiny subset of 100 examples paired with target harmful responses to optimize/identify the safety heads. Evaluation was performed on the full dataset (520 behaviors).
  • StrongREJECT: A dataset of 313 malicious instructions used strictly for out-of-distribution evaluation to test how well the identified safety vectors generalize to unseen harmful scenarios.

Key Insights and Ideas

  • Two Distinct Safety Pathways: Surprisingly, Harmful Patching and Zero Ablation identify completely different sets of attention heads with very low overlap. This indicates that aligned LLMs do not possess a single monolithic safety mechanism. Instead, they maintain separate functional pathways: one that actively suppresses harmful generation and another that is highly sensitive to malicious injections.
Figure 2: Spatial distribution of safety-critical attention heads identified by GOSV. Top: Head locations identified via Zero Ablation. Bottom: Head locations identified via Harmful Patching. Each cell represents an attention head at layer l and position h, with color intensity indicating the selection probability σ(θ(l,h)) after optimization.
  • The Universal 30% Threshold: Across all tested models (Llama-2, Llama-3.1, Qwen2.5, Mistral), a complete safety breakdown occurs when approximately 30% of the total attention heads are repatched. This suggests that safety is encoded in a distributed subset comprising roughly one-third of the model's architecture.
  • Three Phases of Safety Degradation: By progressively patching heads, the authors observed three distinct behavioral phases:
  • Phase I (Dominance): Patching just the top 1% of critical heads causes Attack Success Rate (ASR) to surge from near-zero to over 50%.
  • Phase II (Breakdown): ASR plateaus near 100% as the number of patched heads approaches 30%.
  • Phase III (Dysfunction): Beyond 30%, safety remains broken, but the model's fundamental language capabilities degrade, causing outputs to become incoherent and gibberish.

Key Results

The authors turned these insights into an inference-time white-box jailbreak attack. By simply replacing the activations at the globally optimized head locations, they achieved state-of-the-art jailbreak performance without altering model weights or engineering adversarial prompt suffixes.

  • Near-Perfect Jailbreaks: The attack achieved a 100% success rate on Mistral-7B and 96.35% on Llama-3.1-8B on the AdvBench dataset, substantially outperforming classic white-box attacks like GCG, AutoDAN, and AdvPrefix.
  • Superior Generalization: When tested on the out-of-distribution StrongREJECT dataset, GOSV outperformed standard parameter fine-tuning (FT) by margins of 8% to 19%, proving that globally attributed safety vectors capture robust, fundamental safety mechanisms rather than dataset-specific artifacts.

Limitations and Future Work

While GOSV provides deep mechanistic insights, it has a few notable limitations:

  • White-Box Requirement: The attack requires internal access to intermediate activations, meaning it cannot be directly applied to black-box commercial APIs (like GPT-4) without a proxy model.
  • Lack of Defense: The paper focuses heavily on attributing and exploiting these vulnerabilities; the authors have not yet developed a corresponding defense mechanism to harden these attention heads against activation manipulation.
  • Scope: The evaluation was restricted to open-source, English-language models. Extending this analysis to larger proprietary models and multilingual contexts remains an open direction.

Summary updated

Abstract

While Large Language Models (LLMs) are aligned to mitigate risks, their safety guardrails remain fragile against jailbreak attacks. This reveals limited understanding of components governing safety. Existing methods rely on local, greedy attribution that assumes independent component contributions. However, they overlook the cooperative interactions between different components in LLMs, such as attention heads, which jointly contribute to safety mechanisms. We propose \textbf{G}lobal \textbf{O}ptimization for \textbf{S}afety \textbf{V}ector Extraction (GOSV), a framework that identifies safety-critical attention heads through global optimization over all heads simultaneously. We employ two complementary activation repatching strategies: Harmful Patching and Zero Ablation. These strategies identify two spatially distinct sets of safety vectors with consistently low overlap, termed Malicious Injection Vectors and Safety Suppression Vectors, demonstrating that aligned LLMs maintain separate functional pathways for safety purposes. Through systematic analyses, we find that complete safety breakdown occurs when approximately 30\% of total heads are repatched across all models. Building on these insights, we develop a novel inference-time white-box jailbreak method that exploits the identified safety vectors through activation repatching. Our attack substantially outperforms existing white-box attacks across all test models, providing strong evidence for the effectiveness of the proposed GOSV framework on LLM safety interpretability.
BibTeX
@misc{chu2026attributing,
  author = {Fengheng Chu and Jiahao Chen and Yuhong Wang and Jun Wang and Zhihui Fu and Shouling Ji and Songze Li},
  title = {{Attributing and Exploiting Safety Vectors through Global Optimization in Large Language Models}},
  year = {2026},
  eprint = {2601.15801},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2601.15801},
}
Figure