rSIM: Incentivizing Reasoning Capabilities of LLMs via Reinforced Strategy Injection

Sijia Chen, Baochun Li, Di Niu

TL;DR

The researchers introduce rSIM (reinforced Strategy Injection Mechanism), a framework that uses a small "planner" model to guide a "reasoner" LLM by injecting specific reasoning strategies (like self-reflection or decomposition) into its thought process. This approach allows a tiny 0.5B parameter model to outperform models 28 times its size (14B) on complex reasoning tasks, proving that planning and execution are better handled as a multi-agent team.

What Problem Does This Paper Address?

Recent breakthroughs in Reasoning Language Models (RLMs), such as DeepSeek-R1, have shown that reinforcement learning (RL) can induce "aha moments" where models spontaneously learn to self-correct and think deeply. However, there is a "reasoning bottleneck": small LLMs often lack the inherent capacity to discover these strategies on their own. No matter how much RL you throw at a 0.5B model, it may never "stumble upon" the concept of sub-planning or validation.

The authors address this gap by asking: Can we explicitly guide a model's reasoning process by injecting human-designed strategies, and can we train a separate agent to decide exactly when to use them?

Methods

The authors propose a two-agent system based on a leader-follower framework:

  1. The Planner (Leader): A small LLM that observes the current state of the problem and selects one of nine human-designed strategies (e.g., Self-Reflection, Decomposition, Deep Thinking, Validation) to inject as a prompt.
  2. The Reasoner (Follower): The base LLM that generates the next step of the "Chain-of-Thought" (CoT) based on the planner's instruction.

Key Technical Innovations: * Multi-Agent RL (MARL): Both agents are jointly trained using a modified version of Group Relative Policy Optimization (GRPO). The planner is rewarded for selecting strategies that lead to correct answers, while the reasoner is rewarded for following the plan and solving the problem. * Two-Stage Training: To avoid "conflicting updates" where the two agents confuse each other, the authors use a weighting parameter ($\lambda$). The first stage prioritizes optimizing the planner, while the second stage focuses on the reasoner. * Pluggability: Once trained, the planner can be used as a "plug-in" for other models (even those it wasn't trained with) to immediately boost their reasoning performance without further fine-tuning.

Datasets

The framework was evaluated across seven diverse datasets:

  • Mathematics: MATH, GSM8K, and the highly challenging AIME2024.
  • Multi-task Reasoning: MMLU-Pro and TheoremQA.
  • Code Generation: CodeAlpaca-20k and HumanEval.

Key Insights and Ideas

  • Strategy Frequency Matters: The authors found a direct correlation between how often a model uses reasoning strategies and its final accuracy. Small models fail because their "strategy count" is effectively zero.
  • Decoupling Planning from Reasoning: By separating the decision of how to think from the act of thinking, rSIM allows small models to focus on execution while the planner handles the high-level logic.
  • The "Aha" Moment can be Injected: Instead of waiting for a model to discover self-reflection through millions of RL steps, rSIM "injects" the ability, allowing even 0.5B models to exhibit advanced RLM behaviors.
  • Cross-Model Generalization: A planner trained on a Qwen model can successfully guide a Llama model. This suggests that "good planning" is a universal skill that transcends specific model architectures.

Key Results

  • Punching Above Weight: A Qwen2.5-0.5B model trained with rSIM achieved accuracy on the MATH dataset comparable to or exceeding a Qwen2.5-14B model.
  • SOTA for Small Models: The Qwen2.5-7B + rSIM configuration achieved state-of-the-art results, even outperforming massive models like Llama-3.1-405B-Inst on certain benchmarks.
  • Continual Learning: A planner initially trained on math could be further fine-tuned on coding tasks (CodeAlpaca), resulting in a 17% to 24.4% accuracy boost on HumanEval while still retaining its math skills.
  • Llama Boost: Adding an rSIM planner to Llama3.2-1B brought its performance remarkably close to the much larger Llama3.3-70B.

Limitations and Future Work

  • Fixed Strategy Set: The current system relies on nine human-defined strategies. The planner cannot currently "invent" new reasoning strategies that humans haven't thought of.
  • Imbalanced Preferences: The authors noted that the planner tends to over-rely on "Self-Reflection" and "Decomposition," sometimes neglecting other useful strategies like "Validation" or "Prioritization."
  • Prompt Sensitivity: The effectiveness of the system depends on the quality of the natural language prompts used to describe the strategies.
  • Future Direction: The authors suggest moving toward a dynamic strategy space where the planner can learn to refine or expand its own "toolbox" of reasoning instructions.

Summary updated

Abstract

Large language models (LLMs) are post-trained through reinforcement learning (RL) to evolve into Reasoning Language Models (RLMs), where the hallmark of this advanced reasoning is ``aha'' moments when they start to perform strategies, such as self-reflection and deep thinking, within chain of thoughts (CoTs). Motivated by this, this paper proposes a novel reinforced strategy injection mechanism (rSIM), that enables any LLM to become an RLM by employing a small planner to guide the LLM's CoT through the adaptive injection of reasoning strategies. To achieve this, the planner (leader agent) is jointly trained with an LLM (follower agent) using multi-agent RL (MARL), based on a leader-follower framework and straightforward rule-based rewards. Experimental results show that rSIM enables Qwen2.5-0.5B to become an RLM and significantly outperform Qwen2.5-14B. Moreover, the planner is generalizable: it only needs to be trained once and can be applied as a plug-in to substantially improve the reasoning capabilities of existing LLMs. In addition, the planner supports continual learning across various tasks, allowing its planning abilities to gradually improve and generalize to a wider range of problems.
BibTeX
@inproceedings{chen2026rsim,
  author = {Sijia Chen and Baochun Li and Di Niu},
  title = {{rSIM: Incentivizing Reasoning Capabilities of LLMs via Reinforced Strategy Injection}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.2054},
}
Figure