Cog-Rethinker: Hierarchical Metacognitive Reinforcement Learning for LLM Reasoning

Zexu Sun, Yongcheng Zeng, Erxue Min, Heyang Gao, Bokai Ji, Xu ChenSubmission to ICLR 2026 · Withdrawn

TL;DR

Applying reinforcement learning directly to base language models ("zero-RL") often fails or wastes enormous compute on difficult reasoning tasks because weak models frequently generate zero correct solutions. Cog-Rethinker addresses this by introducing a hierarchical, metacognitive rollout scheme—first prompting models to decompose unsolved problems into subproblems, then prompting them to reflect on and revise failed attempts. Combined with a dynamic demonstration buffer and auxiliary supervised fine-tuning (SFT), Cog-Rethinker drastically boosts sample efficiency and enables even small or weak base models (e.g., Llama 3.2) to overcome training plateaus.

What Problem Does This Paper Address?

Recent breakthroughs in reasoning models (like OpenAI o1 and DeepSeek-R1) leverage Reinforcement Learning with Verifiable Rewards (RLVR) directly on base models. However, standard zero-RL frameworks rely on fixed prompting templates and accuracy filtering (keeping only batches containing both correct and incorrect rollouts to compute policy gradient advantages).

For challenging reasoning tasks or weaker base models (such as smaller 1B–3B parameter models), the probability of finding a correct solution via standard random sampling is nearly zero. When every rollout for a problem fails, no positive reward signal is obtained, leading to wasted training compute, slow convergence, or complete training stagnation where algorithms cannot form valid updates.

Methods

Cog-Rethinker replaces naive direct sampling with a multi-stage rollout pipeline during RL training:

  1. Direct Rollout (DR): The policy generates $G$ standard solutions. If any are correct, the problem proceeds as usual.
  2. Decomposition Rollout (DecR): For zero-accuracy problems, the model is prompted to break the problem into sub-tasks. It references an in-context example retrieved via BM25 from an adaptive Metacognitive Buffer ($M$). The buffer is seeded initially and dynamically updated (FIFO) using the model's own self-generated, successful decompositions.
  3. Reflection Rollout (RefR): For problems that still yield zero correct answers after decomposition, the policy is prompted to examine the previous incorrect solution, locate the error, and generate a revised step-by-step solution.
  4. Policy Optimization with Train-Test Alignment: Rollout samples are trained using DAPO (Decoupled Clip and Dynamic Sampling Policy Optimization). Because inference only uses the standard prompt, all successful solutions from DecR and RefR are mapped back onto the original direct-rollout prompt format. They are then incorporated into training via an auxiliary Supervised Fine-Tuning (SFT) objective ($\mathcal{L}_{\text{SFT}}$) weighted by $\lambda = 0.04$: $$\mathcal{L}_{\text{Cog-Rethinker}}(\theta) = \mathcal{L}_{\text{DAPO}}(\theta) + \lambda \mathcal{L}_{\text{SFT}}(\theta)$$

The authors provide theoretical justification (Theorem 1) showing that problem decomposition and tree-based reflection systematically reduce reward-to-go variance, resulting in strictly lower iteration complexity compared to direct rollout.

       [Problem] 
           │
           ▼
    [Direct Rollout]  ──(Acc > 0)──► Add to Replay Buffer
           │ (Acc = 0)
           ▼
 [Decomposition Rollout] ──(Acc > 0)──► Update Demonstration Buffer & Replay Buffer
           │ (Acc = 0)
           ▼
  [Reflection Rollout]  ──(Acc > 0)──► Add to Replay Buffer
           │
           ▼
  [DAPO + SFT Policy Update] (Rollouts mapped to Direct Prompt)

Datasets

  • Training Dataset: orz-math-127k, an open-source mathematical reasoning training set.
  • Evaluation Benchmarks: Evaluated across nine standard reasoning benchmarks: GSM8K, MATH-500, AIME 2024, AIME 2025, AMC 2023, Gaokao 2023en, Minerva, GPQA-Diamond, and OlympiadBench. No new benchmarks were introduced.

Key Insights and Ideas

  • Overcoming Base Model "Cognitive Boundaries": Increasing raw rollout count under a fixed prompt does not help if the base model lacks the competency to solve a problem directly. Explicit metacognitive prompts (decomposition and error reflection) act as temporary scaffolding, prompting the model to discover reasoning paths it couldn't find through brute-force sampling.
  • Transferring Scaffolding via SFT: Prompting alone causes train-test mismatch if the model requires complex prompts during inference. Training on successful DecR and RefR outputs under the original direct-prompt template bakes structured reasoning into the model's standard response style.
  • Enabling Training on Weak Base Models: In models with limited initial mathematical capability (e.g., Llama 3.2-1B and 3B), online filter-based RL algorithms like BODF and DAPO collapsed because almost zero valid batches were generated. Cog-Rethinker successfully rescued training by creating viable positive trajectories early on.

Key Results

  • Qwen2.5-7B-Base: Cog-Rethinker attained 93.32% on GSM8K (vs. 92.21% for DAPO), 80.60% on MATH-500 (vs. 79.40%), 73.50% on AMC 2023 (vs. 69.00%), and 44.22% on OlympiadBench (vs. 42.44%).
  • Qwen2.5-1.5B-Base: Surpassed DAPO on MATH-500 (59.00% vs. 56.00%) and was the only method to solve problems on AIME 2025 (6.67%, where all other baselines scored 0.00%).
  • Llama 3.2-3B: Both DAPO and BODF failed to train. Cog-Rethinker attained 32.73% on GSM8K and 26.60% on MATH-500, outperforming PPO (20.43% / 17.80%) and GRPO (24.30% / 17.40%).
  • Sample Efficiency: Matched DAPO's rollout budget ($3\times$ rollouts for DAPO to equalize sample budget) still resulted in Cog-Rethinker outperforming it, confirming that performance gains stem from the metacognitive structure rather than simply drawing more tokens.

Limitations and Future Work

  • Dependence on Base Model & Seed Demonstrations: Performance remains tied to the quality of the initial seed demonstrations in the metacognitive buffer and the base model's minimal instruction-following ability.
  • Binary Reward Feedback: The system relies strictly on final-answer correctness; it lacks token- or process-level reward models (PRMs) to evaluate intermediate reasoning quality.
  • Scope: Evaluation is restricted to mathematical and scientific reasoning; the generalizability of decomposition and reflection rollouts to non-mathematical tasks (e.g., open-ended writing, code synthesis) was not explored.

Summary updated

Abstract

Contemporary progress in large language models (LLMs) has revealed notable inferential capacities via reinforcement learning (RL) employing verifiable reward, facilitating the development of O1 and R1-like reasoning models. Directly training from base models with RL is called zero-RL. However, previous works rely upon activating LLMs' inherent capacities through fixed prompt templates. This strategy introduces substantial sampling inefficiencies for weak LLMs, as the majority of problems generate invalid outputs during accuracy-driven filtration in reasoning tasks, which causes a waste of samples. To solve this issue, we propose Cog-Rethinker, a novel hierarchical metacognitive RL framework for LLM reasoning. Our Cog-Rethinker mainly focuses on the rollout procedure in RL training. After the direct rollout, our Cog-Rethinker improves sample utilization in a hierarchical metacognitive two-stage framework. By leveraging human cognition during solving problems, firstly, it prompts policy to decompose zero-accuracy problems into subproblems to produce final reasoning results. Secondly, with zero-accuracy problems in previous rollout stage, it further prompts policy to refine these answers by referencing previous wrong solutions. Moreover, to enable cold-start of the two new reasoning patterns and maintain train-test consistency across prompt templates, our Cog-Rethinker applies supervised fine-tuning on the policy using correct samples of the two stages with direct rollout template. Experimental results demonstrate Cog-Rethinker's superior performance on various mathematical reasoning benchmarks, we also analyzed its improved sample efficiency that accelerates convergence compared to baseline methods.
BibTeX
@misc{sun2025cog,
  author = {Zexu Sun and Yongcheng Zeng and Erxue Min and Heyang Gao and Bokai Ji and Xu Chen},
  title = {{Cog-Rethinker: Hierarchical Metacognitive Reinforcement Learning for LLM Reasoning}},
  year = {2025},
  eprint = {2510.15979},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.15979},
}
Figure