Safety-Utility Conflicts Are Not Global: Surgical Alignment via Head-Level Diagnosis

Wang Cai, Yilin Wen, Jinchang Hou, Du Su, Guoqiu Wang, Zhonghou Lv, Chenfu Bao, Yunfang Wu

Abstract

Safety alignment in Large Language Models (LLMs) often results in an "alignment tax"—a degradation of general capabilities like reasoning and coding. While existing methods apply global fixes, they ignore the modular nature of Transformers. We introduce Conflict-Aware Sparse Tuning (CAST), which identifies specific attention heads where safety and utility goals clash. By selectively updating only "safe" heads and freezing "high-conflict" ones, CAST achieves superior safety alignment with significantly lower performance loss.

The Problem: The Alignment Tax

When LLMs are fine-tuned for safety, they often become less capable at complex tasks. This is typically addressed by treating the model as a homogeneous block and applying global gradient constraints. However, Transformers are composed of specialized modules. Some attention heads are critical for reasoning, while others are more flexible.

Diagnosing Head Conflict and its impact on the Safety-Utility trade-off.

The core issue is that global updates indiscriminately modify "load-bearing" heads where safety requirements and utility goals are in direct opposition. This "surgical" oversight leads to reasoning collapse and over-refusal.

Methodology: Conflict-Aware Sparse Tuning (CAST)

CAST moves away from global updates toward a targeted, head-level approach. The framework consists of three primary stages:

The CAST framework: (A) Diagnosis, (B) Scoring and Zoning, and (C) Surgical Alignment.

1. Head-Level Diagnosis

Before alignment begins, we construct a "Conflict Map" of the frozen base model. For each attention head $h$, we calculate a Conflict Score $C(h)$ derived from:

  • Optimization Conflict ($g_{safe}$ vs $g_{util}$): Measuring the geometric opposition between safety and utility gradients.
  • Functional Sensitivity: Using zero-shot ablation to determine how critical a head is for general utility (Utility Risk) versus its potential for safety improvement (Safety Gain).

2. Risk and Safe Zoning

Heads are ranked by their conflict scores. The top 25% are designated as the "Risky Zone" (high conflict, high sensitivity), while the bottom 25% form the "Safe Zone" (low conflict).

3. Surgical Alignment

During the fine-tuning phase, only the parameters within the Safe Zone heads are updated. The Risky Zone and other intermediate heads remain frozen. This ensures that the model's core reasoning circuits are preserved while safety-aligned behaviors are integrated into more flexible modules.

Key Insights

  • Localized Conflict: Alignment conflicts are not uniform; they are concentrated in specific heads, particularly in the middle-to-deep layers of the network.
  • Predictive Accuracy: The pre-alignment Conflict Score is highly predictive of eventual performance degradation, with a Pearson correlation of $r \geq 0.80$.
  • Efficiency and Performance: Updating just 25% of the heads (the Safe Zone) often yields better safety and utility than full parameter fine-tuning.

Results and Impact

Evaluations on Llama 3.1, Qwen 2.5, and Mistral-7B demonstrate that CAST significantly mitigates the alignment tax.

  • Utility Preservation: On Llama 3.1-8B, CAST improved MMLU accuracy by 9.45% over standard full fine-tuning.
  • Reasoning Recovery: CAST effectively prevents "reasoning collapse" in models like Mistral-7B, where standard methods often cause GSM8K scores to drop sharply.
  • Safety Maintenance: Despite updating fewer parameters, CAST maintains safety levels comparable to or better than global alignment methods.

Conclusion

Safety-utility conflicts in LLMs are a local, modular phenomenon rather than a global one. By diagnosing these conflicts at the head level, CAST provides a parameter-efficient and interpretable path toward safer models that retain their full intellectual capabilities. Future work will extend this diagnosis to MLP layers and explore dynamic conflict mapping during training.

Summary updated

Abstract

Safety alignment in Large Language Models (LLMs) inherently presents a multi-objective optimization conflict, often accompanied by an unintended degradation of general capabilities. Existing mitigation strategies typically rely on global gradient geometry to resolve these conflicts, yet they overlook Modular Heterogeneity within Transformers, specifically that the functional sensitivity and degree of conflict vary substantially across different attention heads. Such global approaches impose uniform update rules across all parameters, often resulting in suboptimal trade-offs by indiscriminately updating utility sensitive heads that exhibit intense gradient conflicts. To address this limitation, we propose Conflict-Aware Sparse Tuning (CAST), a framework that integrates head-level diagnosis with sparse fine-tuning. CAST first constructs a pre-alignment conflict map by synthesizing Optimization Conflict and Functional Sensitivity, which then guides the selective update of parameters. Experiments reveal that alignment conflicts in LLMs are not uniformly distributed. We find that the drop in general capabilities mainly comes from updating a small group of ``high-conflict'' heads. By simply skipping these heads during training, we significantly reduce this loss without compromising safety, offering an interpretable and parameter-efficient approach to improving the safety-utility trade-off.
BibTeX
@misc{cai2026safety,
  author = {Wang Cai and Yilin Wen and Jinchang Hou and Du Su and Guoqiu Wang and Zhonghou Lv and Chenfu Bao and Yunfang Wu},
  title = {{Safety-Utility Conflicts Are Not Global: Surgical Alignment via Head-Level Diagnosis}},
  year = {2026},
  eprint = {2601.04262},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2601.04262},
}
Figure