TL;DR
Reinforcement Learning with Verifiable Rewards (RLVR) helps large language models learn to reason, but its trial-and-error nature makes training highly inefficient. This paper introduces Motivation-enhanced Reinforcement Finetuning (MeRF), a simple method that injects the natural language rules of the reward function directly into the system prompt during training. By letting the model know the "rules of the game," MeRF leverages in-context learning to align generation with optimization, significantly accelerating training and boosting final reasoning performance.
What Problem Does This Paper Address?
State-of-the-art reasoning models (like DeepSeek-R1) frequently leverage RLVR to iteratively refine their cognitive capabilities. In RLVR, models receive objective, automated feedback based on deterministic rules—such as whether a math answer matches the ground truth or if generated code passes unit tests.
However, standard RLVR leaves the model completely blind to the reward function during text generation. The model must blindly explore a sparse reward space, relying entirely on fragmented, trial-and-error feedback to stumble upon successful behaviors. This makes training computationally expensive and leaves the model susceptible to local minima or "reward hacking" (e.g., perfecting output formatting while completely failing the underlying logic).
Methods
The authors introduce MeRF, an intuitive twist on standard reinforcement learning pipelines: they explicitly tell the LLM how it is being evaluated. Because verifiable rewards are rule-based, they can easily be translated into natural language instructions. MeRF directly injects this reward specification (detailing exactly how many points are awarded for correctness vs. formatting) into the system prompt as an in-context motivation.
During the training loop, MeRF utilizes Group Relative Policy Optimization (GRPO), an efficient RL algorithm that optimizes the model by comparing a group of outputs generated from the same prompt, eliminating the need for a resource-heavy critic model. By combining this group-level external reward with the internal, in-context awareness of the objective, the model learns to navigate the solution space far more effectively.
Datasets
MeRF was evaluated across several rigorous reasoning benchmarks:
- Knights and Knaves (K&K) Logic Puzzles: A complex logical deduction dataset used to evaluate scaling and out-of-distribution (OOD) generalization by varying the number of characters in the puzzle.
- MATH Benchmarks: AIME24, AIME25, AMC23, and MATH500 to evaluate competitive mathematical problem-solving.
- CountDown: A numerical reasoning game where the model must construct an equation to reach a target number using a strict set of provided numbers.
- Natural Questions (NQ): A dataset used to test generalizability on an agentic task involving search engine tool usage.
Key Insights and Ideas
- Training Dynamics vs. Test-Time Prompting: Crucially, MeRF's performance boost does not simply come from test-time prompting. The authors demonstrated that validating a standard RLVR-trained model with the motivation prompt only yields minor gains. Instead, having the motivation present during training fundamentally alters and improves the model's exploration path.
- Sustaining Policy Entropy: Standard RLVR models often suffer from rapidly decreasing policy entropy, meaning they quickly collapse into rigid, suboptimal behaviors. MeRF maintains a higher policy entropy throughout training, indicating that understanding the reward space encourages the model to continue exploring diverse reasoning paths effectively.
- Resilience to Misleading Prompts: In a fascinating ablation study, the authors provided the model with an "adverse" motivation prompt (falsely telling it that it would be rewarded for wrong answers, while the actual RL environment still rewarded correct ones). Remarkably, after an initial period of unstable learning, the model learned to discount the misleading prompt and still optimized successfully for the true external reward.
Key Results
- Superior Efficiency and Convergence: MeRF consistently outperformed the RLVR baseline across various model families (Qwen2.5, DeepSeek-R1-Distill) and sizes, demonstrating much faster convergence.
- High Sample Efficiency: On the K&K dataset, MeRF achieved better
pass@4andpass@8performance at training step 140 than the baseline RLVR model achieved at its final step 280.
- Math Performance Gains: On the MATH benchmarks using a Qwen2.5-7B-Base starting point, MeRF achieved consistent improvements over the baseline, yielding an average accuracy gain of up to 4.5% on
pass@8.
Limitations and Future Work
- Static Motivation: The natural language reward descriptions used in MeRF are static throughout the training process. The authors note that exploring dynamic motivations—prompts that adaptively change as the model matures through training—is a promising future direction.
- Dependence on Base Model Capabilities: The approach relies heavily on the LLM's inherent in-context learning ability. For weaker base models that struggle with basic instruction-following, effectively leveraging in-context motivation during RL remains an open challenge.
Summary updated