A Simple "Motivation" Can Enhance Reinforcement Finetuning of Large Reasoning Models

Junjie Zhang, Guozheng Ma, Shunyu Liu, Haoyu Wang, Jiaxing Huang, Ting-En Lin, Fei Huang, Yongbin Li, Dacheng Tao

TL;DR

Reinforcement Learning with Verifiable Rewards (RLVR) helps large language models learn to reason, but its trial-and-error nature makes training highly inefficient. This paper introduces Motivation-enhanced Reinforcement Finetuning (MeRF), a simple method that injects the natural language rules of the reward function directly into the system prompt during training. By letting the model know the "rules of the game," MeRF leverages in-context learning to align generation with optimization, significantly accelerating training and boosting final reasoning performance.

What Problem Does This Paper Address?

State-of-the-art reasoning models (like DeepSeek-R1) frequently leverage RLVR to iteratively refine their cognitive capabilities. In RLVR, models receive objective, automated feedback based on deterministic rules—such as whether a math answer matches the ground truth or if generated code passes unit tests.

However, standard RLVR leaves the model completely blind to the reward function during text generation. The model must blindly explore a sparse reward space, relying entirely on fragmented, trial-and-error feedback to stumble upon successful behaviors. This makes training computationally expensive and leaves the model susceptible to local minima or "reward hacking" (e.g., perfecting output formatting while completely failing the underlying logic).

Methods

The authors introduce MeRF, an intuitive twist on standard reinforcement learning pipelines: they explicitly tell the LLM how it is being evaluated. Because verifiable rewards are rule-based, they can easily be translated into natural language instructions. MeRF directly injects this reward specification (detailing exactly how many points are awarded for correctness vs. formatting) into the system prompt as an in-context motivation.

During the training loop, MeRF utilizes Group Relative Policy Optimization (GRPO), an efficient RL algorithm that optimizes the model by comparing a group of outputs generated from the same prompt, eliminating the need for a resource-heavy critic model. By combining this group-level external reward with the internal, in-context awareness of the objective, the model learns to navigate the solution space far more effectively.

Figure 2: (Left) Illustration of the RLVR pipeline and the in-context motivation introduced by MeRF. Compared to the indirect way (reward samples generated and through parameter updates) to learn the reward patterns, MeRF enables the model to be aware of the overall reward space by in-context motivation. (Right) We validate the Base model, RLVR model and MeRF model on the K&K Logic Puzzle dataset in two settings: w/ motivation and w/o motivation in the prompt. Different from the base model, the RLVR model achieves a slightly better performance in validation w/ motivation than w/o motivation after the RLVR training, even while the motivation is not involved in the training process, indicating a connection between the in-context motivation validation and the RLVR training guided by the reward function (as the motivation describes).

Datasets

MeRF was evaluated across several rigorous reasoning benchmarks:

  • Knights and Knaves (K&K) Logic Puzzles: A complex logical deduction dataset used to evaluate scaling and out-of-distribution (OOD) generalization by varying the number of characters in the puzzle.
  • MATH Benchmarks: AIME24, AIME25, AMC23, and MATH500 to evaluate competitive mathematical problem-solving.
  • CountDown: A numerical reasoning game where the model must construct an equation to reach a target number using a strict set of provided numbers.
  • Natural Questions (NQ): A dataset used to test generalizability on an agentic task involving search engine tool usage.

Key Insights and Ideas

  • Training Dynamics vs. Test-Time Prompting: Crucially, MeRF's performance boost does not simply come from test-time prompting. The authors demonstrated that validating a standard RLVR-trained model with the motivation prompt only yields minor gains. Instead, having the motivation present during training fundamentally alters and improves the model's exploration path.
  • Sustaining Policy Entropy: Standard RLVR models often suffer from rapidly decreasing policy entropy, meaning they quickly collapse into rigid, suboptimal behaviors. MeRF maintains a higher policy entropy throughout training, indicating that understanding the reward space encourages the model to continue exploring diverse reasoning paths effectively.
  • Resilience to Misleading Prompts: In a fascinating ablation study, the authors provided the model with an "adverse" motivation prompt (falsely telling it that it would be rewarded for wrong answers, while the actual RL environment still rewarded correct ones). Remarkably, after an initial period of unstable learning, the model learned to discount the misleading prompt and still optimized successfully for the true external reward.

Key Results

  • Superior Efficiency and Convergence: MeRF consistently outperformed the RLVR baseline across various model families (Qwen2.5, DeepSeek-R1-Distill) and sizes, demonstrating much faster convergence.
Figure 1: Validation Accuracy of MeRF and RLVR baseline on K&K Logic Puzzles in the training process. By simply “telling LLMs rules of the game” with in-context motivation during RL training, MeRF significantly outperforms the RLVR baseline with faster improvements, demonstrating the effectiveness of leveraging in-context motivation for more efficient RL training of LLMs.
  • High Sample Efficiency: On the K&K dataset, MeRF achieved better pass@4 and pass@8 performance at training step 140 than the baseline RLVR model achieved at its final step 280.
Figure 3: Pass@k performance of MeRF and RLVR baseline during the training process (from 0 to 280 steps) on K&K Logic Puzzle. We compare the pass@1, pass@2, pass@4, and pass@8 performance at each step, where MeRF consistently outperforms the RLVR baseline in all metrics. More importantly, MeRF demonstrates a significant training efficiency over RLVR baseline, for example, achieving better pass@4 and pass@8 performance at step 140 than the final RLVR model (at step 280), while RLVR’s performance of pass@4 and pass@8 hardly improves after step 140.
  • Math Performance Gains: On the MATH benchmarks using a Qwen2.5-7B-Base starting point, MeRF achieved consistent improvements over the baseline, yielding an average accuracy gain of up to 4.5% on pass@8.

Limitations and Future Work

  • Static Motivation: The natural language reward descriptions used in MeRF are static throughout the training process. The authors note that exploring dynamic motivations—prompts that adaptively change as the model matures through training—is a promising future direction.
  • Dependence on Base Model Capabilities: The approach relies heavily on the LLM's inherent in-context learning ability. For weaker base models that struggle with basic instruction-following, effectively leveraging in-context motivation during RL remains an open challenge.

Summary updated

Abstract

Reinforcement Learning with Verifiable Rewards~(RLVR) has emerged as a powerful learn-to-reason paradigm for large reasoning models to tackle complex tasks. However, the current RLVR paradigm is still not efficient enough, as it works in a trial-and-error manner. To perform better, the model needs to explore the reward space by numerously generating responses and learn from fragmented reward signals, blind to the overall reward patterns. Fortunately, verifiable rewards make the natural language description of the reward function possible, and meanwhile, LLMs have demonstrated strong in-context learning ability. This motivates us to explore if large reasoning models can benefit from a \textbf{motivation} of the task, \textit{i.e.}, awareness of the reward function, during the reinforcement finetuning process, as we humans sometimes do when learning. In this paper, we introduce \textit{\textbf{M}otivation-\textbf{e}nhanced \textbf{R}einforcement \textbf{F}inetuning}~(\textbf{MeRF}), an intuitive yet effective method enhancing reinforcement finetuning of LLMs by involving \emph{``telling LLMs rules of the game''}. Specifically, \textbf{MeRF} directly injects the reward specification into the prompt, which serves as an in-context motivation for the model to be aware of the optimization objective. This simple modification leverages the in-context learning ability of LLMs, aligning generation with optimization, thereby incentivizing the model to generate desired outputs from both inner motivation and external reward. Empirical evaluations demonstrate that \textbf{MeRF} achieves substantial performance gains over the RLVR baseline. Moreover, ablation studies show that MeRF performs better with greater consistency between the in-context motivation and the external reward function, while the model also demonstrates an ability to adapt to misleading motivations through reinforcement finetuning.
BibTeX
@inproceedings{zhang2026simple,
  author = {Junjie Zhang and Guozheng Ma and Shunyu Liu and Haoyu Wang and Jiaxing Huang and Ting-En Lin and Fei Huang and Yongbin Li and Dacheng Tao},
  title = {{A Simple "Motivation" Can Enhance Reinforcement Finetuning of Large Reasoning Models}},
  booktitle = {International Conference on Learning Representations},
  year = {2026},
  url = {https://openreview.net/forum?id=3owSlsYDQf},
}
Figure