Safety Recovery in Reasoning Models Is Only a Few Early Steering Steps Away

Soumya Suvra Ghosal, Souradip Chakraborty, Vaibhav Singh, Furong Huang, Dinesh Manocha, Amrit Singh Bedi

Abstract

Reinforcement learning (RL) based post-training for explicit chain-of-thought (e.g., GRPO) improves the reasoning ability of multimodal large-scale reasoning models (MLRMs). But recent evidence shows that it can simultaneously degrade safety alignment and increase jailbreak success rates. We propose SafeThink, a lightweight inference-time defense that treats safety recovery as a satisficing constraint rather than a maximization objective. SafeThink monitors the evolving reasoning trace with a safety reward model and conditionally injects an optimized short corrective prefix ("Wait, think safely") only when the safety threshold is violated. In our evaluations across six open-source MLRMs and four jailbreak benchmarks (JailbreakV-28K, Hades, FigStep, and MM-SafetyBench), SafeThink reduces attack success rates by 30-60% (e.g., LlamaV-o1: 63.33% to 5.74% on JailbreakV-28K, R1-Onevision: 69.07% to 5.65% on Hades) while preserving reasoning performance (MathVista accuracy: 65.20% to 65.00%). A key empirical finding from our experiments is that safety recovery is often only a few steering steps away: intervening in the first 1-3 reasoning steps typically suffices to redirect the full generation toward safe completions.

Introduction

The recent surge in "thinking" AI models, such as DeepSeek-R1 and OpenAI's o1, has demonstrated the power of reinforcement learning (RL) in enhancing complex reasoning through explicit chain-of-thought (CoT) traces. While these models excel at math, logic, and coding, this performance boost often comes with a "reasoning tax": a significant degradation in safety alignment. As these models engage in deep reasoning, they become more vulnerable to adversarial jailbreaks, particularly in multimodal settings where harmful instructions can be embedded within images.

The Problem: Safety Degradation in MLRMs

RL fine-tuning for reasoning often causes "conditional coverage collapse." While the model's underlying safety knowledge is not erased, the RL process biases the model to prioritize helpfulness and reasoning depth over safety guardrails. Under adversarial conditions, the model assigns near-zero probability to safe thoughts. Existing defenses, such as static system prompts or truncating the reasoning process, are often ineffective or overly restrictive, significantly harming the model's utility.

The SafeThink Framework

We propose SafeThink, an inference-time steering mechanism designed to recover safety without sacrificing reasoning capabilities. SafeThink is built on the principle of satisficing—aiming for safety that is "good enough" to meet a threshold, rather than attempting to maximize safety at the expense of performance.

SafeThink operates through two main steps:

  1. Monitoring: As the model generates its reasoning trace, an external safety reward model (such as Llama-Guard-3) evaluates the partial output.
  2. Conditional Steering: If the safety score drops below a predefined threshold, the system intervenes by injecting a pre-optimized, short steering token—specifically the phrase "Wait, think safely"—into the context window.

This intervention nudges the model's internal state back toward its latent safety alignment. The steering phrase was selected through an offline optimization process to maximize the probability of safe continuations while minimizing divergence from the model's natural reasoning style.

Key Insights and Results

Our evaluation across six open-source MLRMs and multiple safety benchmarks (JailbreakV-28K, HADES, FigStep, and MM-SafetyBench) yielded several critical insights:

  • Massive Safety Gains: SafeThink reduced Attack Success Rates (ASR) by 30–60% across the board. For instance, on the JailbreakV-28K benchmark, the ASR for LlamaV-o1 dropped from 63.33% to 5.74%. On the HADES benchmark, R1-Onevision's ASR fell from 69.07% to 5.65%.
  • Early Steering is Sufficient: A key finding is that safety recovery is often only a few steps away. Intervening in just the first 1 to 3 reasoning steps is usually enough to redirect the entire generation toward a safe path. Once the initial intent is steered, the model's internal logic naturally sustains the safe trajectory.
  • Preserving Utility: Unlike heavy-handed defenses, SafeThink maintains core reasoning performance. Average MathVista accuracy across models remained stable (63.51% original vs. 63.46% with SafeThink), proving that safety and utility are not mutually exclusive.
  • Low Overhead: Because steering typically occurs early in the process, the inference latency overhead is minimal, adding only 0.1 to 0.9 seconds per query.

Conclusion

SafeThink demonstrates that the safety vulnerabilities introduced by RL-based reasoning training can be mitigated through targeted, inference-time interventions. By treating safety as a satisficing constraint and focusing on early-stage steering, we can effectively recover safety guardrails in multimodal reasoning models without compromising their advanced problem-solving abilities.

Summary updated

Abstract

Reinforcement learning (RL) based post-training for explicit chain-of-thought (e.g., GRPO) improves the reasoning ability of multimodal large-scale reasoning models (MLRMs). But recent evidence shows that it can simultaneously degrade safety alignment and increase jailbreak success rates. We propose SafeThink, a lightweight inference-time defense that treats safety recovery as a satisficing constraint rather than a maximization objective. SafeThink monitors the evolving reasoning trace with a safety reward model and conditionally injects an optimized short corrective prefix ("Wait, think safely") only when the safety threshold is violated. In our evaluations across six open-source MLRMs and four jailbreak benchmarks (JailbreakV-28K, Hades, FigStep, and MM-SafetyBench), SafeThink reduces attack success rates by 30-60% (e.g., LlamaV-o1: 63.33% to 5.74% on JailbreakV-28K, R1-Onevision: 69.07% to 5.65% on Hades) while preserving reasoning performance (MathVista accuracy: 65.20% to 65.00%). A key empirical finding from our experiments is that safety recovery is often only a few steering steps away: intervening in the first 1-3 reasoning steps typically suffices to redirect the full generation toward safe completions.
BibTeX
@inproceedings{ghosal2026safety,
  author = {Soumya Suvra Ghosal and Souradip Chakraborty and Vaibhav Singh and Furong Huang and Dinesh Manocha and Amrit Singh Bedi},
  title = {{Safety Recovery in Reasoning Models Is Only a Few Early Steering Steps Away}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=rCbbDqQzai},
}
Figure