Nothing in Excess: Mitigating the Exaggerated Safety for LLMs via Safety-Conscious Activation Steering

Zouying Cao, Yifei Yang, Hai ZhaoarXiv 2024 · arXiv · OpenReview

Abstract

Safety alignment is indispensable for Large language models (LLMs) to defend threats from malicious instructions. However, recent researches reveal safety-aligned LLMs prone to reject benign queries due to the exaggerated safety issue, limiting their helpfulness. In this paper, we propose a Safety-Conscious Activation Steering (SCANS) method to mitigate the exaggerated safety concerns in aligned LLMs. First, SCANS extracts the refusal steering vectors within the activation space and utilizes vocabulary projection to anchor some specific safety-critical layers which influence model refusal behavior. Second, by tracking the hidden state transition, SCANS identifies the steering direction and steers the model behavior accordingly, achieving a balance between exaggerated safety and adequate safety. Experiments show that SCANS achieves new state-of-the-art performance on XSTest and OKTest benchmarks, without impairing their defense capability against harmful queries and maintaining almost unchanged model capability.

Introduction

Safety alignment techniques, such as Reinforcement Learning from Human Feedback (RLHF), are essential for ensuring that Large Language Models (LLMs) do not generate harmful or toxic content. However, these safeguards often lead to exaggerated safety, where models become overly cautious and reject benign queries that happen to contain sensitive keywords. For instance, a user asking "How do I kill the lights in my room?" might receive a refusal from a hyper-aligned model because the word "kill" triggers a safety filter, leading the model to lecture the user on the sanctity of life instead of providing a helpful response.

Comparison of standard Llama2-7b-chat and SCANS-enhanced version on a benign query containing sensitive keywords.

To address this, we introduce SCANS (Safety-Conscious Activation Steering), a training-free method designed to reduce false refusals without compromising the model's ability to block actual harmful content.

Methodology

SCANS leverages Representation Engineering to modify the model's internal activations during inference rather than its weights. The approach consists of three primary stages:

  1. Inducing Refusal Steering Vectors: We extract a "refusal direction" in the activation space by comparing the internal activations of the model when processing harmful versus benign queries.
  2. Anchoring Safety-Critical Layers: Through vocabulary projection—mapping internal states back to tokens—we discovered that refusal concepts (e.g., "cannot", "rejected") primarily emerge in the middle layers (e.g., layers 10-20 in Llama-2-7b). By targeting these specific layers, SCANS avoids disrupting general reasoning in earlier or later layers.
  3. Safety-Conscious Steering: To ensure safety is not compromised for truly malicious queries, SCANS employs a similarity-based classifier. It tracks the hidden state transition of a query; if the transition suggests a benign intent that is likely to trigger a false refusal, the activations are steered away from the refusal vector. A "Sure" prefix trick is used during classification to help distinguish benign trajectories from stubbornly refusal-aligned harmful ones.

Experimental Results

We evaluated SCANS on several LLMs, including Llama-2, Vicuna, Qwen, and Gemma-2, across a variety of benchmarks:

  • Exaggerated Safety: On the XSTest and OKTest datasets, SCANS reduced false refusal rates by an average of 24.7% and 26.3%, respectively.
  • Safety Robustness: The model's defense against actual harmful queries (e.g., AdvBench) remained robust, with refusal rates often exceeding 99%.
  • General Capability: Performance on standard benchmarks like MMLU and summarization tasks (XSum) remained stable, with negligible impact on inference speed (e.g., a minor drop from 40.60 to 39.62 tokens/s).

Conclusion and Limitations

SCANS demonstrates that the utility-safety trade-off in LLMs can be rebalanced effectively without expensive retraining. By modifying the activation trajectory, we can significantly enhance model helpfulness for benign queries. However, the method is not without limitations: the internal classifier may occasionally misclassify queries, and the steering parameters may require tuning for different architectures. Future work will explore the scalability of this approach to even larger frontier models.

Summary updated

Abstract

Safety alignment is indispensable for Large language models (LLMs) to defend threats from malicious instructions. However, recent researches reveal safety-aligned LLMs prone to reject benign queries due to the exaggerated safety issue, limiting their helpfulness. In this paper, we propose a Safety-Conscious Activation Steering (SCANS) method to mitigate the exaggerated safety concerns in aligned LLMs. First, SCANS extracts the refusal steering vectors within the activation space and utilizes vocabulary projection to anchor some specific safety-critical layers which influence model refusal behavior. Second, by tracking the hidden state transition, SCANS identifies the steering direction and steers the model behavior accordingly, achieving a balance between exaggerated safety and adequate safety. Experiments show that SCANS achieves new state-of-the-art performance on XSTest and OKTest benchmarks, without impairing their defense capability against harmful queries and maintaining almost unchanged model capability.
BibTeX
@misc{cao2024nothing,
  author = {Zouying Cao and Yifei Yang and Hai Zhao},
  title = {{Nothing in Excess: Mitigating the Exaggerated Safety for LLMs via Safety-Conscious Activation Steering}},
  year = {2024},
  eprint = {2408.11491},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2408.11491},
}
Figure