TL;DR
Current safety-trained LLMs often suffer from "over-refusal," turning down harmless requests that superficially resemble unsafe topics. This paper introduces SEAR (Safety Exploration through Adversarial Reasoning), a framework that trains a single model to deliberately explore unsafe interpretations during its internal reasoning phase, only to reliably "flip back" to a safe, helpful response in the final answer. By using segment-level process rewards to balance these competing objectives, SEAR significantly reduces over-refusal while remaining highly robust against adversarial attacks.
What Problem Does This Paper Address?
When we train Large Language Models (LLMs) to be safe, we often make them overly cautious. If a user asks an ambiguous question—for example, asking about a medical condition or a sensitive historical event—the model might preemptively refuse to answer out of an abundance of caution. This is known as over-refusal.
While recent techniques use Reinforcement Learning (RL) to encourage models to "reason" before answering, the authors find that this reasoning often acts as a mere "rubber stamp" for a predetermined refusal. Current models typically exhibit two brittle behaviors: they either avoid unsafe interpretations entirely (leading to over-refusal), or they explore unsafe thoughts but lose control and output a harmful final answer.
The authors argue that safety reasoning should mimic math or coding reasoning: a model should be allowed to explore a "wrong" or unsafe path internally to resolve ambiguity, recognize the boundary, and then self-correct to deliver a safe and helpful final response.
Methods
To teach a model to explore dangerous thoughts safely, the authors frame safety training as an adversarial optimization problem with competing rewards within a single Chain-of-Thought (CoT) rollout:
- The Reasoning Segment ($z$): Rewarded for being harmful or unsafe ($r_h$), encouraging the model to fully explore the malicious intent or risky interpretations of a prompt.
- The Answer Segment ($y$): Rewarded for being safe ($r_s$), ensuring the final output is completely aligned and secure.
Optimizing these opposing goals within one model is notoriously unstable. Standard outcome-based RL applies a single scalar reward to the entire trajectory, which fails because answer safety optimizes much faster than reasoning harmfulness, causing the model's behavior to degenerate.
To solve this, the authors utilize process-based rewards via Group Relative Policy Optimization (GRPO). They split the advantage calculations: tokens in the reasoning segment are evaluated solely on the harmfulness reward, while tokens in the answer segment are evaluated on the safety reward.
Additionally, because competing objectives cause oscillatory training dynamics, the authors apply iterate averaging, training four independent runs and averaging their weights to produce the final SEAR-1.5B model.
Datasets
The model was trained and evaluated using a comprehensive suite of datasets:
- Training Data: A 1:1 mix of harmful prompts (WildJailbreak, Aegis, SafeEdit) and harmless instruction tasks (UltraFeedback mixed with 10% OR-Bench).
- Safety Evaluation (x-axis): HarmBench prompts subjected to adversarial jailbreak attacks like PAIR and GCG.
- Compliance/Over-Refusal Evaluation (y-axis): XSTest, Fortress (CBRN domain), and False-Reject to measure how often the model correctly answers benign but sensitive prompts.
- Out-of-Distribution (OOD) Generalization: GSM8K, MATH-500, GPQA, MMLU-Pro, and SQuAD.
Key Insights and Ideas
- Unsafe Reasoning as Exploration: Allowing a model to think about the "bad" interpretation of a prompt actually helps it map out safety boundaries more clearly, allowing it to fulfill benign parts of ambiguous requests instead of panicking and refusing.
- The Failure of Outcome Rewards: Standard dense outcome rewards cause a mismatch in convergence rates. Answer safety saturates early, leaving the reasoning reward to dominate and bleed harmful thinking into completely benign tasks. Decoupling credit assignment via process rewards is mandatory.
- Resilience to Pre-filling: Because SEAR is explicitly trained to recover from harmful reasoning chains, it is uniquely robust against "pre-fill attacks"—where an attacker forces the model to start its response with a malicious sentence to hijack its safety guardrails.
Key Results
- Optimal Safety-Refusal Trade-off: SEAR-1.5B achieved the best point on the Pareto frontier with a quadratic mean score of 0.930 (where 1.0 is perfect safety and compliance), outperforming standard outcome RL (0.892) and even the much larger TARS-7B model (0.909).
- Controlled Flip-Back Behavior: As shown in the training metrics, process rewards successfully increased reasoning exploration without letting final answer safety degrade.
- Pre-fill Attack Defense: When forced by an adversarial pre-fill to generate harmful reasoning, SEAR-1.5B maintained an answer safety score of 0.901 on ClearHarm, drastically outperforming the base Qwen3-1.7B model (0.610).
- Preserved Capabilities: SEAR maintained its out-of-distribution capabilities on general reasoning and math benchmarks better than standard outcome-based RL baselines.
Limitations and Future Work
The primary limitation noted by the authors is their specific choice of the dense process reward, which directly incentivizes the most harmful path during the reasoning phase. They note that other, more nuanced forms of exploratory reasoning might mitigate over-refusal even more effectively without explicitly leaning into maximum harmfulness. Future work includes extending this dense-reward framework to alternative reward formulations for the reasoning segment.
Summary updated