Uncovering Safety Risks of Large Language Models through Concept Activation Vector

Zhihao Xu, Ruixuan Huang, Changyu Chen, Xiting Wang

TL;DR

This paper introduces the Safety Concept Activation Vector (SCAV) framework, revealing that large language models (LLMs) represent the concept of "safety" linearly within their intermediate hidden layers. By leveraging this insight, the authors develop a highly efficient, data-frugal jailbreak attack method that achieves near-perfect attack success rates on open-source models and transfers effectively to black-box APIs like GPT-4.

What Problem Does This Paper Address?

Despite intensive safety alignment efforts (such as Reinforcement Learning from Human Feedback), LLMs remain stubbornly vulnerable to jailbreak attacks. Existing white-box attacks—which manipulate the internal states of a model—often rely on crude heuristics or time-consuming grid searches to determine where and how much to perturb model embeddings.

This lack of a principled optimization goal frequently results in low-quality, repetitive, or garbled text generation. Furthermore, these methods often demand substantial training data. The authors address this gap by accurately modeling how LLMs internally represent safety, allowing for automated, precise, and high-quality attacks at both the embedding and prompt levels.

Methods

The core innovation is the Safety Concept Activation Vector (SCAV) framework. The authors train a simple linear classifier on the hidden states of an LLM's intermediate layers to distinguish between safe and malicious prompts.

  • Linear Interpretability: The authors demonstrate that for aligned models, safety becomes linearly separable after the 10th or 11th layer.
  • Embedding-Level Attack: Instead of guessing a perturbation vector heuristically, they solve a constrained optimization problem to find the minimum distance needed to push a malicious embedding across the decision boundary into the "safe" subspace. This yields a clean, closed-form mathematical solution for the optimal perturbation magnitude and direction.
  • Prompt-Level Attack: For black-box scenarios where internal parameters are hidden, they use the SCAV classifier to guide a hierarchical genetic algorithm (adapted from AutoDAN). This optimizes token sequences to minimize the model's internal perception of maliciousness, bypassing the need to guess a hardcoded target response string (e.g., "Sure, here is how to...").

Datasets

The authors evaluate their approach using several standard safety benchmarks:

  • Advbench and HarmfulQA: Used to gather malicious instructions for training (140 prompts) and testing (50 prompts).
  • StrongREJECT: A rigorous evaluation dataset containing 50 diverse malicious scenarios.
  • Harmbench: An independent dataset containing 80 cases used to validate embedding-level attacks.
  • AdvExtent: Employed to test the efficacy of model unlearning methods.
  • Note: Safe instructions used for training the classifier were generated using GPT-4.

Key Insights and Ideas

  • Safety is a Linear Concept: Aligned LLMs consistently cluster safe and malicious instructions into linearly separable spaces in their middle-to-late layers. Unaligned models do not exhibit this behavior. Linear separability of safety across layers
Figure 1: Test accuracy of Pm on different layers of LLMs.
  • The Shortest Path to Jailbreak: Because the safety boundary is linear, the most effective perturbation direction is simply perpendicular to the separating hyperplane. This avoids the suboptimal or counterproductive directions generated by heuristic baselines like RepE or JRE. Comparison of perturbation directions
Figure 2: Comparison of perturbations added by our method (SCAV) and the baselines RepE [16] and JRE [18]. Our method consistently moves embeddings of malicious instructions to the subspace of safe instructions, while the baselines may result in ineffective or even opposite perturbations.
  • Model Self-Correction: If an attacker perturbs only a single layer, subsequent layers will actually "correct" the embedding and steer the model back toward a refusal. To successfully jailbreak a model, the attack must perturb a sequence of intermediate layers (e.g., layers 10–13) to permanently override the safety mechanism.
  • "Unlearning" is an Illusion: Defense methods designed to make LLMs "unlearn" harmful knowledge (like Eraser) do not actually delete the information. Instead, they just shift the safety classification boundary, which SCAV easily bypasses.

Key Results

  • Near-Perfect Success: SCAV achieved an average attack success rate (ASR) of 99.14% across seven popular open-source LLMs (including LLaMA-2, LLaMA-3, and Mistral) using standard keyword matching.
  • Extreme Data Efficiency: While other representation engineering methods require large datasets, SCAV requires as few as 5 pairs of malicious and safe instructions to train an effective classifier.
  • Black-Box Transferability: Prompt-level attacks optimized via SCAV successfully transferred to GPT-4, inducing useful responses to malicious prompts up to 84% of the time on Advbench.
  • High Response Quality: Human and GPT-4 evaluations showed that SCAV dramatically reduced language flaws (like repetition or gibberish) by 16% to 42% compared to baseline attacks, maintaining the model's natural text generation capabilities.

Limitations and Future Work

The authors note that while their method is highly effective at executing both embedding-level and prompt-level attacks, they lack an in-depth exploration of the underlying transferability mechanisms—specifically, the theoretical reasons why a perturbation vector optimized for one white-box model can successfully exploit another. They highlight the exploration of these transfer mechanisms as a vital direction for future work in building more robust AI defenses.

Summary updated

Abstract

Despite careful safety alignment, current large language models (LLMs) remain vulnerable to various attacks. To further unveil the safety risks of LLMs, we introduce a Safety Concept Activation Vector (SCAV) framework, which effectively guides the attacks by accurately interpreting LLMs' safety mechanisms. We then develop an SCAV-guided attack method that can generate both attack prompts and embedding-level attacks with automatically selected perturbation hyperparameters. Both automatic and human evaluations demonstrate that our attack method significantly improves the attack success rate and response quality while requiring less training data. Additionally, we find that our generated attack prompts may be transferable to GPT-4, and the embedding-level attacks may also be transferred to other white-box LLMs whose parameters are known. Our experiments further uncover the safety risks present in current LLMs. For example, in our evaluation of seven open-source LLMs, we observe an average attack success rate of 99.14%, based on the classic keyword-matching criterion. Finally, we provide insights into the safety mechanism of LLMs. The code is available at https://github.com/SproutNan/AI-Safety_SCAV.
BibTeX
@inproceedings{xu2024uncovering,
  author = {Zhihao Xu and Ruixuan Huang and Changyu Chen and Xiting Wang},
  title = {{Uncovering Safety Risks of Large Language Models through Concept Activation Vector}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=Uymv9ThB50},
}
Figure