Uncovering Safety Risks in Open-source LLMs through Concept Activation Vector

Zhihao Xu, Ruixuan Huang, Xiting Wang, Fangzhao Wu, Jing Yao, Xing Xie

Abstract

Despite careful safety alignment, current large language models (LLMs) remain vulnerable to various attacks. To further unveil the safety risks of LLMs, we introduce a Safety Concept Activation Vector (SCAV) framework, which effectively guides the attacks by accurately interpreting LLMs' safety mechanisms. We then develop an SCAV-guided attack method that can generate both attack prompts and embedding-level attacks with automatically selected perturbation hyperparameters. Both automatic and human evaluations demonstrate that our attack method significantly improves the attack success rate and response quality while requiring less training data. Additionally, we find that our generated attack prompts may be transferable to GPT-4, and the embedding-level attacks may also be transferred to other white-box LLMs whose parameters are known. Our experiments further uncover the safety risks present in current LLMs. For example, we find that six out of seven open-source LLMs that we attack consistently provide relevant answers to more than 85% malicious instructions. Finally, we provide insights into the safety mechanism of LLMs.

Introduction

The rapid advancement of Large Language Models (LLMs) has been accompanied by rigorous safety alignment efforts, such as Reinforcement Learning from Human Feedback (RLHF). However, these models remain susceptible to "jailbreak" attacks—carefully crafted prompts designed to bypass safety filters and elicit harmful content. Existing attack methodologies often rely on black-box trial-and-error or complex white-box optimization that may not align with the model's internal logic. This research addresses the gap in understanding LLMs' internal safety mechanisms by introducing the Safety Concept Activation Vector (SCAV) framework, which mathematically defines the "safety concept" within hidden layers to guide more effective attacks.

Methodology: The SCAV Framework

The SCAV framework is rooted in the hypothesis of linear interpretability, suggesting that high-level concepts like "safety" or "maliciousness" are represented as specific directions within the model's high-dimensional embedding space.

  1. Defining the Safety Direction: A simple linear classifier is trained on the hidden states of an LLM to distinguish between malicious and safe instructions. The vector perpendicular to the resulting decision boundary is defined as the SCAV.
  2. Embedding-Level Attack: For white-box models, SCAV enables a precise perturbation of internal embeddings. By calculating a closed-form solution, the attack moves a malicious embedding across the safety boundary into the "safe" subspace via the shortest possible path.
  3. Prompt-Level Attack: In black-box scenarios, the SCAV classifier guides a genetic algorithm. The search objective is to minimize the model's internal "maliciousness" score, rather than merely targeting specific keyword-based responses.
SCAV framework visualization showing how malicious embeddings are moved to the safe subspace via the shortest path.

Key Insights and Ideas

Experimental analysis using the SCAV framework revealed several critical properties of LLM safety:

  • Safety as a "Late Bloomer": The internal representation of safety is not present in the initial layers of the model. As illustrated in the layer-wise accuracy analysis, the ability to linearly separate safe and malicious inputs only matures significantly around the 10th or 11th layer.
  • Extreme Data Efficiency: Due to the clear linear definition of the safety concept, the SCAV attack requires minimal training data. Peak performance can be achieved with as few as five pairs of malicious and safe instructions.
  • Fragility of Unlearning: Tests on models using "unlearning" defenses showed that SCAV could still elicit harmful responses with a 97.34% success rate, suggesting these defenses may only obscure rather than delete harmful knowledge.
Test accuracy of the safety classifier across different layers, showing safety emergence in later layers.

Results and Evaluation

The SCAV-guided attack was evaluated against baselines like GCG and AutoDAN across seven popular open-source LLMs, including LLaMA-3 and Mistral.

  • High Success Rates: The method achieved an average attack success rate (ASR) of 99.14% across open-source models.
  • Improved Response Quality: Human evaluations indicated an 8-10% improvement in the usefulness and coherence of harmful responses compared to previous state-of-the-art methods.
  • Cross-Model Transferability: Prompts optimized for LLaMA models demonstrated significant transferability, achieving up to an 84% ASR on GPT-4, which suggests shared internal safety representations across different model architectures.

Conclusion

The SCAV framework provides a powerful lens for interpreting and exploiting the safety mechanisms of LLMs. By demonstrating that safety is represented as a robust linear direction in embedding space, this work highlights the vulnerabilities of current alignment and unlearning techniques. Future research should focus on developing defenses that operate within these internal embedding spaces to provide more resilient protection against sophisticated attacks.

Summary updated

Abstract

Despite careful safety alignment, current large language models (LLMs) remain vulnerable to various attacks. To further unveil the safety risks of LLMs, we introduce a Safety Concept Activation Vector (SCAV) framework, which effectively guides the attacks by accurately interpreting LLMs' safety mechanisms. We then develop an SCAV-guided attack method that can generate both attack prompts and embedding-level attacks with automatically selected perturbation hyperparameters. Both automatic and human evaluations demonstrate that our attack method significantly improves the attack success rate and response quality while requiring less training data. Additionally, we find that our generated attack prompts may be transferable to GPT-4, and the embedding-level attacks may also be transferred to other white-box LLMs whose parameters are known. Our experiments further uncover the safety risks present in current LLMs. For example, we find that six out of seven open-source LLMs that we attack consistently provide relevant answers to more than 85\% malicious instructions. Finally, we provide insights into the safety mechanism of LLMs.
BibTeX
@inproceedings{xu2024uncovering,
  author = {Zhihao Xu and Ruixuan Huang and Xiting Wang and Fangzhao Wu and Jing Yao and Xing Xie},
  title = {{Uncovering Safety Risks in Open-source LLMs through Concept Activation Vector}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=Uymv9ThB50},
}
Figure