Abstract: While reinforcement learning-trained Large Reasoning Models (LRMs, e.g., Deepseek-R1) demonstrate advanced reasoning capabilities in the evolving Large Language Models (LLMs) domain, their susceptibility to security threats remains a critical vulnerability. This weakness is particularly evident in Chain-of-Thought (CoT) generation processes, where adversarial methods like backdoor prompt attacks can systematically subvert the model's core reasoning mechanisms. The emerging Chain-of-Thought Attack (CoTA) reveals this vulnerability through exploiting prompt controllability, simultaneously degrading both CoT safety and task performance with low-cost interventions. To address this compounded security-performance vulnerability, we propose Thought Purity (TP): a defense framework that systematically strengthens resistance to malicious content while preserving operational efficacy. Our solution achieves this through three synergistic components: (1) a safety-optimized data processing pipeline (2) reinforcement learning-enhanced rule constraints (3) adaptive monitoring metrics. Our approach establishes the first comprehensive defense mechanism against CoTA vulnerabilities in reinforcement learning-aligned reasoning systems, significantly advancing the security-functionality equilibrium for next-generation AI architectures.
Introduction
Large Reasoning Models (LRMs) like OpenAI’s o1 and DeepSeek-R1 have introduced a paradigm shift by utilizing Chain-of-Thought (CoT) reasoning to solve complex tasks. However, this explicit "thinking out loud" process introduces a novel vulnerability: Chain-of-Thought Attacks (CoTA). In a CoTA scenario, an attacker injects a hidden trigger into a prompt that hijacks the model's internal logic. Instead of a simple refusal or a standard incorrect answer, the model follows a corrupted reasoning path to a malicious conclusion while maintaining a deceptive appearance of coherence.
Traditional defense mechanisms, such as input filtering or refusal-based training, often suffer from an "alignment tax," where increased safety leads to decreased reasoning performance. Moreover, these methods typically fail to disentangle malicious triggers from valid logic, leading to binary outcomes: total compliance with the attack or total refusal of the task.
Methodology
To address these vulnerabilities, we introduce Thought Purity (TP), a defense framework designed to enable "active reasoning recovery." The TP framework consists of two primary components: a specialized data processing pipeline and a reinforcement learning (RL) optimization strategy.
The TP Data Process Pipeline involves generating synthetic CoT data using high-performance teacher models, followed by the intentional injection of "poisoned" triggers. Crucially, we implant safety labels—<suspect> and <harm>—into the training data. These tags teach the model to flag suspicious triggers and isolate corrupted logic, allowing it to proceed to a correct "clean" answer despite the presence of an attack.
The model is then fine-tuned using Group Relative Policy Optimization (GRPO), the same RL algorithm used to train DeepSeek-R1. We employ a dual-reward system: an Outcome Reward (ORM) for final answer accuracy and a Process Reward (PRM) for the correct utilization of safety tags during the reasoning process.
Experimental Evaluation
We evaluated the TP framework across four diverse reasoning benchmarks: Letter (symbolic reasoning), CSQA (commonsense), GSM8K (math), and StrategyQA (multi-step logic). Testing was conducted on several model families, including DeepSeek-R1-Distill-Llama-8B, Qwen3-8B, and Meta-Llama-3.1-8B-Instruct.
Results and Discussion
Our results demonstrate that TP significantly reduces the Attack Success Rate (ASR) while maintaining or even enhancing performance on clean tasks. A key metric introduced is the Cure Rate (CR), which measures the model's ability to recover the correct answer after detecting an attack.
As shown in the results table, TP consistently outperforms baselines in terms of both safety (lower ASR) and recovery (higher CR). For instance, on the StrategyQA dataset with the Qwen3-8B model, the TP method achieved a CR of 13.65%, a significant improvement over original and ORM-only distributions.
Furthermore, our analysis revealed an "Intelligence Paradox": more capable models, which are better at following complex instructions, can paradoxically be more susceptible to CoTA as they more rigorously follow the malicious logic injected by triggers. TP mitigates this by redirecting that instruction-following capability toward self-correction and "thought purification."
Conclusion
Thought Purity represents a fundamental shift from passive refusal to active purification of internal reasoning. By teaching models to identify and isolate malicious logic within their own "internal monologue," we achieve a superior balance between safety and utility. While smaller models (under 8B parameters) show less effectiveness with RL-based defense, the TP paradigm establishes a robust foundation for securing next-generation reasoning architectures against adversarial prompt manipulation.
Summary updated