TL;DR
Aligned LLMs often refuse safe prompts not because of the request's intent, but due to specific "trigger" words. This paper identifies a causal "refusal direction" in activation space and uses it to build a vocabulary-wide "fear lexicon," demonstrating that swapping a "feared" word for a "courageous" synonym (like "poetic" or "whimsical") can significantly mitigate over-refusal.
What Problem Does This Paper Address?
Safety alignment in Large Language Models (LLMs) is a double-edged sword. While it prevents harmful outputs, it often leads to over-refusal: the model declines perfectly benign requests (e.g., "How do I kill a Python process?") because they contain "unsafe-looking" cues.
Previous research suggested that surface-level words act as triggers, but we lacked a way to quantify how much each word matters or a systematic method to fix it without retraining. The authors set out to measure word-level sensitivity across the entire vocabulary and determine if this sensitivity is mediated by a specific, locatable direction in the model's internal representations.
Methods
The researchers employed a multi-step approach across eight instruction-tuned models (including Qwen3.5, Llama-3.2, and Gemma-4):
- Crossed Differencing & ANOVA: To isolate the effect of words from the effect of the prompt, they used a "crossed design." They appended pairs of harm/neutral words (e.g., bomb vs. garden) to various prompts and measured the difference in hidden states at the generation boundary (the last tokens before the model starts generating an answer). They used a vector ANOVA to extract a shared "refusal direction" ($\mu$).
- Activation Steering: To prove causality, they "wrote" along this direction—adding or subtracting the vector from the model's activations during inference.
- Vocabulary Scanning: Using the refusal direction as a ruler, they projected the activations of over 71,000 words to create a "fear table" (lexicon). Words that pushed activations strongly along the refusal direction were labeled "feared," while those pushing the opposite way were labeled "courageous."
- Table-Guided Substitution: They tested if the lexicon could predict behavior by replacing the most "feared" word in a prompt with its most "courageous" meaning-preserving synonym.
Datasets
The study utilized several established benchmarks to test different aspects of refusal:
- OR-Bench (Hard-1K and Toxic): Used for measuring over-refusal on benign prompts and standard refusal on harmful ones.
- XSTest: Provided matched safe and contrast prompts to identify exaggerated safety behaviors.
- WildJailbreak: Used to source a pool of harmful prompts.
- StrongREJECT & IFEval: Used as secondary metrics to ensure that reducing refusal didn't inadvertently increase harmfulness or break instruction-following capabilities.
Key Insights and Ideas
- Refusal is Added Late: Mechanistic analysis on Qwen3.5-4B revealed that while a "harm" word is recognized early, its effect on refusal is primarily written by late MLP layers at the generation boundary.
- Intuition is a Poor Guide: The authors found that "courage" words aren't just generic safe terms like "educational" or "professional." Instead, stylistic "register" words like poetic, whimsical, and sardonic were the most effective at lowering refusal.
- Predictable "Fear": A word's position in the fear lexicon is highly predictable ($ \rho \approx 0.84 $) from its context-free input embedding. This suggests models learn to associate specific regions of the embedding space with safety triggers.
- Shared vs. Specific: While models share a "common core" of feared words (slurs, violent crime terms), each model also has unique lexical sensitivities.
Key Results
- Causal Steering: Writing the refusal direction at a dose of -1.5 removed between 41% and 92% of the refusal caused by inserting a harm word across all eight models.
- The Fear Lexicon: The vocabulary scan produced a clear ranking. At the "feared" end were sexual terms and slurs; at the "courageous" end were whimsical and poetic terms.
- Behavioral Prediction: In held-out tests, replacing the most feared word in a benign prompt with a courageous synonym lowered refusal rates by 3 to 28 percentage points.
- Insertion Impact: Simply appending a harm word to a prompt the model would otherwise answer raised refusal rates by up to 49 points.
Limitations and Future Work
- Model Scale: The study focused on open-weight models up to 27B parameters; it remains to be seen if these findings hold for massive closed models like GPT-4 or Claude 3.
- Language Scope: The current lexicon and analysis are limited to English.
- Dual-Use Risk: The authors acknowledge that the same techniques used to reduce over-refusal could potentially be used to bypass legitimate safety filters, though the effect on harmful prompts was modest (+0.04 rise in harmfulness).
- Future Direction: The authors suggest that safety evaluation should control for vocabulary to be more reliable and that the fear table can serve as a diagnostic tool for developers to audit model oversensitivity.
Summary updated