When we train Large Language Models (LLMs) to be safe, we often accidentally make them unhelpful. Ask a standard safety-aligned model how to "kill a computer process," and it might flatly refuse, mistakenly associating the technical term "kill" with physical violence.
This phenomenon is known as over-refusal bias (or exaggerated safety). It represents a fundamental tension in AI alignment: the safety-utility trade-off. In trying to prevent models from generating harmful content, current alignment techniques often produce models that are overly cautious, rejecting benign requests that superficially resemble dangerous ones.
Why Models Over-Refuse: The Superficial Trigger Problem
To understand why over-refusal happens, we have to look at how models learn safety. During Reinforcement Learning from Human Feedback (RLHF) or Direct Preference Optimization (DPO), a model is penalized for answering harmful prompts and rewarded for refusing them.
However, language models and the reward models guiding them often struggle to differentiate between the semantic intent of a request and its superficial characteristics. As Lambert et al. demonstrate in RewardBench, preference and reward models frequently exhibit a "safety-first bias." They assign high reward scores to any refusal response, regardless of whether the prompt actually violates safety guidelines.
Because the reward model acts as a blunt judge, the conversational model learns a simple heuristic: when in doubt, shut it down. Bianchi et al. show in Safety-Tuned LLaMAs that adding even a small number of safety examples to training data can trigger this exaggerated safety. The model fails to distinguish between truly harmful intent and safe language that shares sensitive keywords, leading to a high rate of false positives.
[User Prompt: "How do I kill this process?"]
│
▼
[Superficial Feature Extraction] ──► Triggers sensitive keyword: "kill"
│
▼
[Blunt Reward Model Judge] ───────► Predicts high reward for ANY refusal
│
▼
[Model Response: "I cannot fulfill this request."] (Over-Refusal)
This vulnerability is not limited to text. In multimodal systems, Xin Liu et al. observe in Safety of Multimodal Large Language Models on Images and Text that over-alignment causes models to reject benign queries simply because a harmless visual element is misidentified as dangerous.
The Blindspot of Context and Perspective
The root cause of over-refusal is a lack of contextual nuance. A prompt that is dangerous in one context may be completely benign—or even vital—in another.
For instance, Sun et al. introduce CASE-Bench to show that commercial models frequently suffer from over-refusal because they ignore professional or educational contexts that render a query safe. Similarly, in specialized domains like medicine, Xin Liu et al. note in VocalAgent that strict, unyielding alignment filters prevent models from assisting with legitimate clinical workflows out of an over-abundance of caution.
Furthermore, over-refusal limits a model's ability to adopt diverse perspectives. When models undergo aggressive safety training, they often begin refusing to adopt specific religious, political, or social personas, failing to function effectively in benign role-playing or analytical scenarios (Kim et al.).
Measuring the Bias
Historically, safety benchmarks focused exclusively on maximizing refusal rates against harmful prompts (e.g., "how to build a bomb"). However, Zeng et al. argue in AIR-BENCH 2024 that optimizing solely for refusal rates actively incentivizes over-refusal.
To accurately evaluate a model's alignment, benchmarks must include safe control prompts. In Do-Not-Answer, Wang et al. demonstrate that without these benign controls, it is impossible to distinguish a robustly secured model from one that is entirely non-functional due to over-sensitivity.
| Evaluation Type | Dataset Composition | What it Measures | Risk of Ignoring Over-Refusal |
|---|---|---|---|
| Traditional Safety | 100% Harmful Prompts | Refusal Rate | Incentivizes a completely broken, non-responsive model. |
| Balanced Safety | Harmful + Benign Control Prompts | False-Refusal Sensitivity | Rewards models that understand context and intent. |
Mitigating Over-Refusal
To break the safety-utility deadlock, researchers are moving away from monolithic refusal training and toward architectures that decouple safety from general capability.
1. Controllable Preference Optimization (CPO)
Instead of forcing a model to permanently compress safety and helpfulness into the same weights, Guo et al. propose Controllable Preference Optimization. By using explicit conditional tokens (such as <Helpfulness:5> or <Harmlessness:5>), the framework allows users to steer objectives dynamically. This separation ensures the model can maximize harmlessness without losing its capacity to provide detailed responses to benign queries.
2. Disentangled Risk Analysis
Rather than evaluating a prompt as a single text string, the DREAM framework by Jianyu Liu et al. forces multimodal models to analyze textual and visual risk factors independently before making a safety judgment. By sharpening the model's internal reasoning about the origin of a risk, it avoids the broad avoidance strategies that cause oversafety.
3. Dynamic Activation and Parameter Steering
Instead of hardcoding refusals during pretraining or fine-tuning, frameworks like EnSToM (Suh et al.) scale behavioral steering vectors dynamically based on the model's internal uncertainty (entropy). This prevents the excessive refusal typically caused by applying fixed-intensity safety steering to ambiguous or edge-case queries.
Ultimately, mitigating over-refusal requires moving from a paradigm of prohibition to one of comprehension. A truly aligned model should not be an easily frightened assistant that plugs its ears at sensitive keywords, but a sophisticated interlocutor capable of evaluating intent, context, and nuance.