TL;DR
Applying reinforcement learning directly to base language models ("zero-RL") often fails or wastes enormous compute on difficult reasoning tasks because weak models frequently generate zero correct solutions. Cog-Rethinker addresses this by introducing a hierarchical, metacognitive rollout scheme—first prompting models to decompose unsolved problems into subproblems, then prompting them to reflect on and revise failed attempts. Combined with a dynamic demonstration buffer and auxiliary supervised fine-tuning (SFT), Cog-Rethinker drastically boosts sample efficiency and enables even small or weak base models (e.g., Llama 3.2) to overcome training plateaus.
What Problem Does This Paper Address?
Recent breakthroughs in reasoning models (like OpenAI o1 and DeepSeek-R1) leverage Reinforcement Learning with Verifiable Rewards (RLVR) directly on base models. However, standard zero-RL frameworks rely on fixed prompting templates and accuracy filtering (keeping only batches containing both correct and incorrect rollouts to compute policy gradient advantages).
For challenging reasoning tasks or weaker base models (such as smaller 1B–3B parameter models), the probability of finding a correct solution via standard random sampling is nearly zero. When every rollout for a problem fails, no positive reward signal is obtained, leading to wasted training compute, slow convergence, or complete training stagnation where algorithms cannot form valid updates.
Methods
Cog-Rethinker replaces naive direct sampling with a multi-stage rollout pipeline during RL training:
- Direct Rollout (DR): The policy generates $G$ standard solutions. If any are correct, the problem proceeds as usual.
- Decomposition Rollout (DecR): For zero-accuracy problems, the model is prompted to break the problem into sub-tasks. It references an in-context example retrieved via BM25 from an adaptive Metacognitive Buffer ($M$). The buffer is seeded initially and dynamically updated (FIFO) using the model's own self-generated, successful decompositions.
- Reflection Rollout (RefR): For problems that still yield zero correct answers after decomposition, the policy is prompted to examine the previous incorrect solution, locate the error, and generate a revised step-by-step solution.
- Policy Optimization with Train-Test Alignment: Rollout samples are trained using DAPO (Decoupled Clip and Dynamic Sampling Policy Optimization). Because inference only uses the standard prompt, all successful solutions from DecR and RefR are mapped back onto the original direct-rollout prompt format. They are then incorporated into training via an auxiliary Supervised Fine-Tuning (SFT) objective ($\mathcal{L}_{\text{SFT}}$) weighted by $\lambda = 0.04$: $$\mathcal{L}_{\text{Cog-Rethinker}}(\theta) = \mathcal{L}_{\text{DAPO}}(\theta) + \lambda \mathcal{L}_{\text{SFT}}(\theta)$$
The authors provide theoretical justification (Theorem 1) showing that problem decomposition and tree-based reflection systematically reduce reward-to-go variance, resulting in strictly lower iteration complexity compared to direct rollout.
[Problem]
│
▼
[Direct Rollout] ──(Acc > 0)──► Add to Replay Buffer
│ (Acc = 0)
▼
[Decomposition Rollout] ──(Acc > 0)──► Update Demonstration Buffer & Replay Buffer
│ (Acc = 0)
▼
[Reflection Rollout] ──(Acc > 0)──► Add to Replay Buffer
│
▼
[DAPO + SFT Policy Update] (Rollouts mapped to Direct Prompt)
Datasets
- Training Dataset:
orz-math-127k, an open-source mathematical reasoning training set. - Evaluation Benchmarks: Evaluated across nine standard reasoning benchmarks: GSM8K, MATH-500, AIME 2024, AIME 2025, AMC 2023, Gaokao 2023en, Minerva, GPQA-Diamond, and OlympiadBench. No new benchmarks were introduced.
Key Insights and Ideas
- Overcoming Base Model "Cognitive Boundaries": Increasing raw rollout count under a fixed prompt does not help if the base model lacks the competency to solve a problem directly. Explicit metacognitive prompts (decomposition and error reflection) act as temporary scaffolding, prompting the model to discover reasoning paths it couldn't find through brute-force sampling.
- Transferring Scaffolding via SFT: Prompting alone causes train-test mismatch if the model requires complex prompts during inference. Training on successful DecR and RefR outputs under the original direct-prompt template bakes structured reasoning into the model's standard response style.
- Enabling Training on Weak Base Models: In models with limited initial mathematical capability (e.g., Llama 3.2-1B and 3B), online filter-based RL algorithms like BODF and DAPO collapsed because almost zero valid batches were generated. Cog-Rethinker successfully rescued training by creating viable positive trajectories early on.
Key Results
- Qwen2.5-7B-Base: Cog-Rethinker attained 93.32% on GSM8K (vs. 92.21% for DAPO), 80.60% on MATH-500 (vs. 79.40%), 73.50% on AMC 2023 (vs. 69.00%), and 44.22% on OlympiadBench (vs. 42.44%).
- Qwen2.5-1.5B-Base: Surpassed DAPO on MATH-500 (59.00% vs. 56.00%) and was the only method to solve problems on AIME 2025 (6.67%, where all other baselines scored 0.00%).
- Llama 3.2-3B: Both DAPO and BODF failed to train. Cog-Rethinker attained 32.73% on GSM8K and 26.60% on MATH-500, outperforming PPO (20.43% / 17.80%) and GRPO (24.30% / 17.40%).
- Sample Efficiency: Matched DAPO's rollout budget ($3\times$ rollouts for DAPO to equalize sample budget) still resulted in Cog-Rethinker outperforming it, confirming that performance gains stem from the metacognitive structure rather than simply drawing more tokens.
Limitations and Future Work
- Dependence on Base Model & Seed Demonstrations: Performance remains tied to the quality of the initial seed demonstrations in the metacognitive buffer and the base model's minimal instruction-following ability.
- Binary Reward Feedback: The system relies strictly on final-answer correctness; it lacks token- or process-level reward models (PRMs) to evaluate intermediate reasoning quality.
- Scope: Evaluation is restricted to mathematical and scientific reasoning; the generalizability of decomposition and reflection rollouts to non-mathematical tasks (e.g., open-ended writing, code synthesis) was not explored.
Summary updated