Abstract
Safety alignment in Large Language Models (LLMs) often results in an "alignment tax"—a degradation of general capabilities like reasoning and coding. While existing methods apply global fixes, they ignore the modular nature of Transformers. We introduce Conflict-Aware Sparse Tuning (CAST), which identifies specific attention heads where safety and utility goals clash. By selectively updating only "safe" heads and freezing "high-conflict" ones, CAST achieves superior safety alignment with significantly lower performance loss.
The Problem: The Alignment Tax
When LLMs are fine-tuned for safety, they often become less capable at complex tasks. This is typically addressed by treating the model as a homogeneous block and applying global gradient constraints. However, Transformers are composed of specialized modules. Some attention heads are critical for reasoning, while others are more flexible.
The core issue is that global updates indiscriminately modify "load-bearing" heads where safety requirements and utility goals are in direct opposition. This "surgical" oversight leads to reasoning collapse and over-refusal.
Methodology: Conflict-Aware Sparse Tuning (CAST)
CAST moves away from global updates toward a targeted, head-level approach. The framework consists of three primary stages:
1. Head-Level Diagnosis
Before alignment begins, we construct a "Conflict Map" of the frozen base model. For each attention head $h$, we calculate a Conflict Score $C(h)$ derived from:
- Optimization Conflict ($g_{safe}$ vs $g_{util}$): Measuring the geometric opposition between safety and utility gradients.
- Functional Sensitivity: Using zero-shot ablation to determine how critical a head is for general utility (Utility Risk) versus its potential for safety improvement (Safety Gain).
2. Risk and Safe Zoning
Heads are ranked by their conflict scores. The top 25% are designated as the "Risky Zone" (high conflict, high sensitivity), while the bottom 25% form the "Safe Zone" (low conflict).
3. Surgical Alignment
During the fine-tuning phase, only the parameters within the Safe Zone heads are updated. The Risky Zone and other intermediate heads remain frozen. This ensures that the model's core reasoning circuits are preserved while safety-aligned behaviors are integrated into more flexible modules.
Key Insights
- Localized Conflict: Alignment conflicts are not uniform; they are concentrated in specific heads, particularly in the middle-to-deep layers of the network.
- Predictive Accuracy: The pre-alignment Conflict Score is highly predictive of eventual performance degradation, with a Pearson correlation of $r \geq 0.80$.
- Efficiency and Performance: Updating just 25% of the heads (the Safe Zone) often yields better safety and utility than full parameter fine-tuning.
Results and Impact
Evaluations on Llama 3.1, Qwen 2.5, and Mistral-7B demonstrate that CAST significantly mitigates the alignment tax.
- Utility Preservation: On Llama 3.1-8B, CAST improved MMLU accuracy by 9.45% over standard full fine-tuning.
- Reasoning Recovery: CAST effectively prevents "reasoning collapse" in models like Mistral-7B, where standard methods often cause GSM8K scores to drop sharply.
- Safety Maintenance: Despite updating fewer parameters, CAST maintains safety levels comparable to or better than global alignment methods.
Conclusion
Safety-utility conflicts in LLMs are a local, modular phenomenon rather than a global one. By diagnosing these conflicts at the head level, CAST provides a parameter-efficient and interpretable path toward safer models that retain their full intellectual capabilities. Future work will extend this diagnosis to MLP layers and explore dynamic conflict mapping during training.
Summary updated