TL;DR
Refusal in large language models (LLMs) isn't a single binary switch, but a "register" that can be precisely steered. By decomposing the refusal direction in Gemma-2 models, researchers found that different categories of refusal (e.g., harmful content vs. self-reference) occupy distinct activation directions, allowing them to change how a model refuses—switching its style from "As an AI..." to "This is dangerous"—without changing the decision to refuse itself.
What Problem Does This Paper Address?
When an LLM refuses a prompt, it usually does so for one of several reasons: the request is harmful (e.g., "how to build a bomb"), or it's a "meta" query about its own internal workings (e.g., "show me your system prompt").
Previous research suggested that refusal is mediated by a single "refusal direction" in the model's internal activations. If you ablate (delete) this direction, the model stops refusing. However, this paper asks a more nuanced question: Does this internal machinery only decide whether to refuse, or does it also control the register—the specific vocabulary and style—of the refusal?
The authors aim to move beyond the "one direction, one mechanism" view to see if refusal is a mixture of category-aligned components that play distinct functional roles.
Methods
The researchers focused on Gemma-2-9B-IT (and its smaller 2B sibling), using Sparse Autoencoders (SAEs) from the Gemma Scope project to look under the hood.
- Direction Finding: They used Linear Support Vector Machines (SVMs) to identify two distinct directions in the residual stream at Layer 20:
- $B_1$: Separates harmful content from neutral prompts.
- $B_2$: Separates self-reference queries from neutral prompts.
- Interventions:
- Ablation: They zeroed out the top-K SAE features aligned with these directions to see if the model would stop refusing specific categories.
- Steering (Sufficiency): They "clamped" these features to high values to see if they could force the model to refuse benign questions.
- Direction Injection: This is the "headline" experiment. They added the $B_1$ (harmful) vector to the activations of $B_2$ (self-reference) prompts. The goal was to see if they could make the model refuse a system-prompt query using the "scary" language typically reserved for dangerous content.
- Measurement: They used regex-based lexicons to count "hits" for two styles:
- Content Register: Words like "harmful," "illegal," or "dangerous."
- Meta Register: Phrases like "as a language model" or "I'm just an AI."
Datasets
The study used small, highly targeted prompt sets (10 for fitting, 10–20 for evaluation) across three categories:
- HARMFUL: Requests for synthesizing controlled substances or weapons.
- NORMAL: Factual, benign questions (e.g., "What is the speed of light?").
- SELFREF: Queries about the model's identity or system prompt (e.g., "What are your internal rules?").
Key Insights and Ideas
- Refusal is Multi-Dimensional: The $B_1$ and $B_2$ directions are roughly 75° apart. They are not the same thing. Refusal is a "concept cone" rather than a single line.
- Causal Steering of Style: You can change the flavor of a refusal. By injecting the "harmful" direction into a "self-reference" prompt, the model still refuses, but it stops saying "I'm an AI" and starts saying "I cannot fulfill your request... it could be used to harm others."
- The Asymmetry of Refusal: $B_1$ (harmful) is a "heavyweight" direction. It encodes both the detection of the category and the specific vocabulary to use. $B_2$ (self-ref) is "lighter"—it seems to primarily detect the category and then hand off the actual language generation to downstream layers.
- Depth Matters: The effects of these directions evolve through the layers. $B_2$ effects are detectable as early as Layer 9, while $B_1$ is most potent around Layer 20. By Layer 31, the directions seem to encode the act of refusing (the language production) rather than the reason for refusing.
Key Results
- Register Shift: Injecting the $B_1$ direction into self-reference prompts increased the "content-safety" vocabulary rate from 2.55 to 10.37 hits per 100 tokens. Random control directions only produced 1.13–2.16 hits.
- Dissociable Ablation: Ablating top-250 $B_1$ features reduced harmful refusal to 55% but left self-reference refusal at 100%. Conversely, ablating $B_2$ features dropped self-reference refusal to 35% while harmful refusal stayed at 95%.
- Model Scaling: The findings were qualitatively replicated in the Gemma-2-2B-IT model, suggesting this is a consistent architectural property of the Gemma-2 family.
Limitations and Future Work
- Lexicon Limitations: The use of regex-based scoring is a bit rigid. Future work could use LLM judges to better capture the nuance of "register."
- Small Sample Size: The prompt sets were small (n=10-20). While the results were statistically significant, larger-scale testing is needed.
- Cross-Model Generalization: The study only looked at Gemma. It remains to be seen if Llama, Mistral, or Qwen exhibit the same asymmetric, category-aligned refusal structures.
- Practical Safety: The authors note that jailbreak benchmarks should report per-category metrics. Defeating the "harmful" refusal mechanism might leave the "self-reference" mechanism completely intact, giving a false sense of a "total" jailbreak.
Summary updated