Metacognition as Reward: Reinforcing LLM Reasoning via Knowledge and Regulation Signals

Sirui Chen, Lei Xu, Yuying Zhao, Yutian Chen, Yu Wang, Beier Zhu, Hanwang Zhang, Shengjie Zhao, Chaochao LuarXiv 2026 · arXiv

Here is a concise, engaging summary of the paper "Metacognition as Reward: Reinforcing LLM Reasoning via Knowledge and Regulation Signals".

TL;DR

The paper introduces Metacognition-as-Reward (MaR), a reinforcement learning framework that trains Large Language Models (LLMs) to think about how they think. By structuring and rewarding intermediate reasoning steps based on metacognitive knowledge and regulation, MaR delivers substantial performance gains across 22 benchmarks, allowing a 9B parameter model to outperform models more than ten times its size.

What Problem Does This Paper Address?

Reinforcement learning (RL) has drastically improved LLM reasoning, but existing reward designs suffer from a goldilocks problem:

  1. Reinforcement Learning with Verifiable Rewards (RLVR): Only checks the final answer (e.g., code execution or math ground truth). This leaves intermediate reasoning weakly constrained—a model can easily get the right answer for the entirely wrong reasons.
  2. Rubrics-as-Reward (RaR): Uses natural language rubrics to judge reasoning quality. However, these rubrics are highly instance-specific, requiring massive human effort to design for every new problem type.

The authors ask a fundamental question: Can we define a general, scalable reward objective for guiding intermediate LLM reasoning across diverse tasks?

To solve this, they turn to cognitive psychology and the concept of metacognition ("cognition about cognition").

Methods

The proposed framework, Metacognition-as-Reward (MaR), forces the model to expose its internal thinking process by scaffolding its output into a structured format, which is then evaluated by an LLM grader.

Figure 1: Overview of MaR. MaR follows a three-stage loop: the policy generates multiple structured rollouts consisting of metacognitive knowledge, metacognitive regulation, and the final answer; a grader scores each rollout along knowledge monitoring, regulation monitoring, and answer correct- ness; and the resulting reward scores are used to optimize the policy.

1. Structured Metacognitive Rollout

When given a prompt, the policy model must generate its response in four distinct stages:

  • Metacognitive Knowledge (MK): The model lists atomic, indexed facts, rules, or constraints relevant to the task.
  • Metacognitive Regulation (MR): The model outlines a step-by-step plan to solve the problem.
  • LOOKBACK (Optional): If the model realizes its initial knowledge was incomplete or it gets stuck, it triggers a recovery step to state what it is seeking and what it found.
  • Final Answer: The actual solution to the problem.
Figure 2: Key components of the rollout gener- ation prompt. The prompt instructs the policy to organize each rollout into MK, MR, an optional LOOKBACK, and the final answer.

2. Metacognitive Monitoring Reward (MMR)

Instead of just checking the final answer, a trajectory-level reward is calculated using three components:

  • Knowledge Monitoring Reward (KMR): Measures how well the initial MK covered the required knowledge, plus any successful recoveries made during the LOOKBACK phase.
  • Regulation Monitoring Reward (RMR): Evaluates how faithfully the model followed its own plan, heavily penalizing "shortcuts" where the model skips its declared steps to guess an answer.
  • Correctness Reward (CR): Evaluates final answer correctness.

The policy is optimized using Direct Advantage Policy Optimization (DAPO), reinforcing trajectories that are both process-aligned and factually correct.

Datasets

The authors evaluated MaR across 22 benchmarks spanning five distinct domains (totaling ~16K test examples):

  • Science: GPQA-Diamond, SuperGPQA, BioProBench, SciCUEval, FrontierScience, ResearchQA.
  • Medical: MedGUIDE, MedMCQA, LongHealth, LLMEval-Med.
  • Long-Context: DocQA-RL-1.6K, LongMIT, LongReward.
  • Math: MMLU-Math, GSM8K, SVAMP, MATH-500, AIME (2024–2026).
  • Logic: FOLIO, ProofWriter.

Training data was derived from existing medical and science datasets, with gold-standard knowledge units synthesized using GPT-5.1 (~32K training samples).

Figure 3: Overview of the evaluation datasets.

Key Insights and Ideas

  • Process Rewards > Outcome Rewards: Explicitly supervising how a model plans and monitors its knowledge leads to better reasoning quality. Vanilla RL methods often caused performance drops on rubric-based tasks because they only optimized for the final answer.
  • Complementary Signals: Statistical analysis showed that Knowledge Monitoring (KMR) and Regulation Monitoring (RMR) capture distinct, non-redundant aspects of the reasoning process, yet both strongly correlate with final correctness.
  • Capacity Matters: Stronger base models (like Qwen3.5-9B vs. 4B) benefit significantly more from metacognitive training, indicating that a baseline level of model capacity is required to internalize complex self-regulation strategies.

Key Results

MaR consistently outperformed base models, vanilla DAPO, and even much larger frontier models:

Table 1: Broad evaluation across science and medical benchmarks. We compare MaR with base models, vanilla DAPO, and frontier models. The table reports average accuracy on each dataset, covering both outcome-based and rubric-based evaluations. Best results in each group are in bold.
  • Performance Boosts: For Qwen3.5-9B, MaR achieved up to a 7.7% absolute accuracy gain over the base model on science tasks and an 11.0% gain over vanilla DAPO on rubric-based benchmarks.
  • David vs. Goliath: Qwen3.5-9B+MaR achieved an overall average score of 67.6%, narrowing the gap to frontier models and successfully surpassing GPT-OSS-120B on average, as well as outperforming Deepseek-V3.2 (685B) on the LongHealth medical document benchmark.
  • Out-of-Domain Generalization: Despite being trained only on science and medical data, MaR-trained models generalized remarkably well to unseen long-context and complex math tasks (e.g., up to a 10.6% gain on the challenging AIME math competition dataset).

Limitations and Future Work

  • Domain Concentration: The training metadata was heavily focused on science and medical tasks. Incorporating broader domains during training could yield even higher generalizability.
  • Text-Only Focus: The current framework is restricted to text-only English reasoning. The authors note that expanding MaR to multilingual, multimodal (vision-language), and interactive agent environments remains an exciting open direction.

Summary updated

Abstract

Recent RL methods have substantially improved the reasoning abilities of LLMs. Existing reward designs mainly follow two paradigms: (1) Reinforcement learning with verifiable rewards (RLVR) derives outcome signals from executable checks or ground-truth answers, but provides limited guidance for intermediate reasoning behaviors. (2) Rubrics-as-reward (RaR) goes beyond final-answer checking by using natural-language rubrics to assess reasoning quality and task compliance, but often requires instance-specific rubrics and substantial design effort. To address these issues, we introduce Metacognition-as-Reward (MaR), a metacognition-inspired RL framework that guides LLM reasoning through two general process dimensions: i) metacognitive knowledge, which identifies task-relevant information without hand-crafted instance-specific rubrics, and ii) metacognitive regulation, which plans and adjusts the reasoning process to provide reward guidance beyond final-answer outcomes. MaR scaffolds model rollouts into explicit metacognitive components and optimizes them with a trajectory-level reward over task knowledge coverage, regulation fidelity, and final-answer correctness. In this way, MaR extends reward feedback to reasoning trajectories while grounding the reward signals in general metacognitive dimensions. Experiments on 22 benchmarks show that MaR consistently improves model performance, achieving up to a 7.7% gain over the base model and up to an 11.0% gain over vanilla DAPO. Notably, Qwen3.5-9B + MaR narrows the gap to frontier models, surpassing GPT-OSS-120B on overall average and outperforming stronger models on several individual benchmarks. Process-level analysis further shows substantial improvements in reasoning process quality. MaR also generalizes to out-of-domain datasets, where MaR-trained models improve over their corresponding base models on average.
BibTeX
@misc{chen2026metacognition,
  author = {Sirui Chen and Lei Xu and Yuying Zhao and Yutian Chen and Yu Wang and Beier Zhu and Hanwang Zhang and Shengjie Zhao and Chaochao Lu},
  title = {{Metacognition as Reward: Reinforcing LLM Reasoning via Knowledge and Regulation Signals}},
  year = {2026},
  eprint = {2605.23384},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2605.23384},
}

Loading related content...

Figure