Addressing Over-Refusal in LLMs with Competing Rewards

Taeyoun Kim, Aviral KumararXiv 2026 · arXiv

TL;DR

Current safety-trained LLMs often suffer from "over-refusal," turning down harmless requests that superficially resemble unsafe topics. This paper introduces SEAR (Safety Exploration through Adversarial Reasoning), a framework that trains a single model to deliberately explore unsafe interpretations during its internal reasoning phase, only to reliably "flip back" to a safe, helpful response in the final answer. By using segment-level process rewards to balance these competing objectives, SEAR significantly reduces over-refusal while remaining highly robust against adversarial attacks.

What Problem Does This Paper Address?

When we train Large Language Models (LLMs) to be safe, we often make them overly cautious. If a user asks an ambiguous question—for example, asking about a medical condition or a sensitive historical event—the model might preemptively refuse to answer out of an abundance of caution. This is known as over-refusal.

While recent techniques use Reinforcement Learning (RL) to encourage models to "reason" before answering, the authors find that this reasoning often acts as a mere "rubber stamp" for a predetermined refusal. Current models typically exhibit two brittle behaviors: they either avoid unsafe interpretations entirely (leading to over-refusal), or they explore unsafe thoughts but lose control and output a harmful final answer.

The authors argue that safety reasoning should mimic math or coding reasoning: a model should be allowed to explore a "wrong" or unsafe path internally to resolve ambiguity, recognize the boundary, and then self-correct to deliver a safe and helpful final response.

Figure 2: Harmful Reasoning & Flip Back Count. Harmfulness of generated reasoning on 1000 WildJailbreak (Jiang et al., 2024) harmful prompts categorized into 5 buckets. Top Row: most frontier models rarely explore harmful reasoning traces and are mostly safe. Bottom Row: as the reasoning becomes harmful, the answer becomes less safe, showing a lack of capability to flip back. Results for SEAR-1.5B are in provided in Figure 5.

Methods

To teach a model to explore dangerous thoughts safely, the authors frame safety training as an adversarial optimization problem with competing rewards within a single Chain-of-Thought (CoT) rollout:

  1. The Reasoning Segment ($z$): Rewarded for being harmful or unsafe ($r_h$), encouraging the model to fully explore the malicious intent or risky interpretations of a prompt.
  2. The Answer Segment ($y$): Rewarded for being safe ($r_s$), ensuring the final output is completely aligned and secure.

Optimizing these opposing goals within one model is notoriously unstable. Standard outcome-based RL applies a single scalar reward to the entire trajectory, which fails because answer safety optimizes much faster than reasoning harmfulness, causing the model's behavior to degenerate.

To solve this, the authors utilize process-based rewards via Group Relative Policy Optimization (GRPO). They split the advantage calculations: tokens in the reasoning segment are evaluated solely on the harmfulness reward, while tokens in the answer segment are evaluated on the safety reward.

Additionally, because competing objectives cause oscillatory training dynamics, the authors apply iterate averaging, training four independent runs and averaging their weights to produce the final SEAR-1.5B model.

Figure 1: Reward Comparison. Comparison of outcome-based, dense outcome-based, and process- based rewards. Outcome-based rewards use the final answer reward (rs) across all generated tokens. Dense outcome-based rewards combine the reward from the answer (rs) and reasoning (rh) into all tokens. Process-based rewards, which SEAR uses, separate the answer (rs) and reasoning (rh) rewards into its corresponding tokens.

Datasets

The model was trained and evaluated using a comprehensive suite of datasets:

  • Training Data: A 1:1 mix of harmful prompts (WildJailbreak, Aegis, SafeEdit) and harmless instruction tasks (UltraFeedback mixed with 10% OR-Bench).
  • Safety Evaluation (x-axis): HarmBench prompts subjected to adversarial jailbreak attacks like PAIR and GCG.
  • Compliance/Over-Refusal Evaluation (y-axis): XSTest, Fortress (CBRN domain), and False-Reject to measure how often the model correctly answers benign but sensitive prompts.
  • Out-of-Distribution (OOD) Generalization: GSM8K, MATH-500, GPQA, MMLU-Pro, and SQuAD.

Key Insights and Ideas

  • Unsafe Reasoning as Exploration: Allowing a model to think about the "bad" interpretation of a prompt actually helps it map out safety boundaries more clearly, allowing it to fulfill benign parts of ambiguous requests instead of panicking and refusing.
  • The Failure of Outcome Rewards: Standard dense outcome rewards cause a mismatch in convergence rates. Answer safety saturates early, leaving the reasoning reward to dominate and bleed harmful thinking into completely benign tasks. Decoupling credit assignment via process rewards is mandatory.
  • Resilience to Pre-filling: Because SEAR is explicitly trained to recover from harmful reasoning chains, it is uniquely robust against "pre-fill attacks"—where an attacker forces the model to start its response with a malicious sentence to hijack its safety guardrails.

Key Results

  • Optimal Safety-Refusal Trade-off: SEAR-1.5B achieved the best point on the Pareto frontier with a quadratic mean score of 0.930 (where 1.0 is perfect safety and compliance), outperforming standard outcome RL (0.892) and even the much larger TARS-7B model (0.909).
  • Controlled Flip-Back Behavior: As shown in the training metrics, process rewards successfully increased reasoning exploration without letting final answer safety degrade.
Figure 3: Training metrics and trends. Comparison of dense outcome rewards, filtering, and process rewards with the same training objective. (a) the answer safety on harmful prompts remains the same across the three methods, (b) the reasoning harmfulness on harmful prompts explodes for dense reward without filtering and decreases when filtering with outcome rewards, (c) the answer helpfulness on harmless prompts decreases without filtering and with outcome-based rewards, (d) the uncontrolled increase of reasoning harmfulness on harmful prompts without filtering also increases reasoning harmfulness on harmless prompts.
  • Pre-fill Attack Defense: When forced by an adversarial pre-fill to generate harmful reasoning, SEAR-1.5B maintained an answer safety score of 0.901 on ClearHarm, drastically outperforming the base Qwen3-1.7B model (0.610).
  • Preserved Capabilities: SEAR maintained its out-of-distribution capabilities on general reasoning and math benchmarks better than standard outcome-based RL baselines.

Limitations and Future Work

The primary limitation noted by the authors is their specific choice of the dense process reward, which directly incentivizes the most harmful path during the reasoning phase. They note that other, more nuanced forms of exploratory reasoning might mitigate over-refusal even more effectively without explicitly leaning into maximum harmfulness. Future work includes extending this dense-reward framework to alternative reward formulations for the reasoning segment.

Summary updated

Abstract

Safety training on language models often induces over-refusal: improved safety on harmful prompts at the cost of increased refusal on harmless ones. Though this trade-off can be mitigated by training models with reinforcement learning (RL) to reason before answering, it does not remove the underlying problem that reasoning can often be a "rubber stamp" for a predetermined response. In this paper, we address the safety-refusal trade-off by rethinking how models are trained to reason about safety. Our key insight is that unsafe reasoning can itself serve as a useful exploratory signal. Rather than preemptively blocking harmful thoughts, we encourage the model to sufficiently explore unsafe reasoning but produce a safe response. The harmful exploration improves the model's ability to distinguish harmful from harmless prompts by resolving ambiguity, allowing it to remain safe while complying only when appropriate. We cast this as an adversarial optimization problem in which a reasoning player explores strategies for producing an unsafe response and an answer player ensures that the final output is safe. We train a single model with dense rewards to play both roles within one chain-of-thought, across different segments. To achieve this, we find that process rewards are crucial for stable optimization of competing objectives. Our resulting model SEAR deliberately engages in harmful reasoning as exploration while reliably flipping back to a safe answer. We demonstrate that this behavior helps mitigate over-refusal and defend against attacks that directly manipulate the reasoning to be harmful.
BibTeX
@misc{kim2026addressing,
  author = {Taeyoun Kim and Aviral Kumar},
  title = {{Addressing Over-Refusal in LLMs with Competing Rewards}},
  year = {2026},
  eprint = {2606.31748},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2606.31748},
}
Figure