Margin-Guided Token Credit for Reinforcement Learning with Verifiable Rewards

Chi Liu, Xin Chen, Yuchuan Weng, Srinivasan ManoharanarXiv 2026 · arXiv

TL;DR

Researchers at PayPal.AI have introduced MCPO (Margin-Credit Policy Optimization), a simple yet effective way to improve how language models learn to reason. By focusing reinforcement learning rewards on "close call" tokens—where the model hesitated between two likely candidates—MCPO significantly boosts performance on competition-math benchmarks compared to standard GRPO, all without adding any extra computational overhead.

What Problem Does This Paper Address?

In Reinforcement Learning with Verifiable Rewards (RLVR), a model is rewarded if it gets the final answer to a math problem right. The current standard, GRPO (Group Relative Policy Optimization), takes that final reward and spreads it equally across every single token in the solution.

This is fundamentally inefficient. In a 3,000-token derivation, tokens like "Let" or "=" are rarely in doubt and don't deserve much credit for a correct answer. Conversely, if the model makes a mistake, punishing these trivial tokens doesn't help it learn where the logic actually failed. While some researchers have tried using token entropy to locate these critical "reasoning forks," entropy can be a noisy signal—a distribution with one clear favorite and a long tail of unlikely options can still have high entropy even if the next step isn't actually a difficult decision.

Methods

The authors propose a narrower, more targeted metric: the top-2 margin ($M = p_1 - p_2$). This measures the gap between the most likely next token and the runner-up. A small margin indicates a "close call" where the model was genuinely uncertain.

The MCPO Workflow:

  1. Measure Hesitation: During the standard training forward pass, the model calculates the top-2 margin for every token.
  2. Score Uncertainty: This margin is mapped to a score $h$ using fixed thresholds. If the margin is $\geq 0.9$, the score is 0 (confident). If it's $\leq 0.3$, the score is 1 (uncertain).
  3. Redistribute Credit: The sequence-level advantage (how much better/worse this solution was than others in its group) is redistributed across tokens based on these scores.
  4. In a positive completion: Uncertain tokens receive more of the reward.
  5. In a negative completion: Uncertain tokens receive less of the penalty, meaning confident tokens carry more of the blame for the failure.
Figure 1: Overview. (a) Relative to the GRPO baseline, the method changes only the token-level advantages; the new components are orange. The old-policy forward pass that the trainer already runs also gives the top-2 margin M = p1 −p2 at every position, and a fixed-threshold map turns it into a score h ∈[0, 1]. (b) An illustrative completion. Tokens with a clear winner get h = 0 and close calls get h ≈1. In a positive completion the close calls receive more of the advantage; in a negative one they receive less of the penalty. The mean token advantage stays Ai.

Crucially, because the trainer already performs a forward pass to calculate importance ratios, the top-2 probabilities are "free." MCPO requires no extra model evaluations or complex batch statistics.

Datasets

  • Training: 2,048 problems sampled from DAPO-Math-17k.
  • Evaluation: Six competition-math benchmarks totaling 297 problems: AIME24, AIME25, AIME26, HMMT26, AMC23, and MATH500 level 5.

Key Insights and Ideas

  • Margin vs. Entropy: The authors argue that the margin is a more precise tool for finding reasoning forks than entropy. As shown in Figure 2 (below), distributions with the same margin can vary wildly in entropy, but the margin specifically captures the "closeness" of the top decision.
  • Sign-Aware Weighting: By shielding uncertain tokens from the full weight of a penalty in failed attempts, the model is less likely to "prune" complex reasoning paths. This encourages the model to keep exploring difficult steps rather than just shortening its answers to avoid mistakes.
  • The "Length" Effect: One of the most striking findings was that MCPO models write significantly longer answers (2-3x longer than the baseline). The authors hypothesize that by rewarding "close calls," the model learns to expand its reasoning at critical junctures.
Comparison of entropy vs. top-2 margin, showing that margin is a more specific indicator of a "close call."

Key Results

MCPO consistently outperformed the standard GRPO baseline across different models and data seeds:

  • MiniCPM5-2B-Base: Achieved a mean gain of +6.55 points in macro avg@32.
  • Qwen3-4B-Base: Achieved a mean gain of +4.75 points.
  • Scaling with Budget: The advantage of MCPO grows with the sampling budget. At pass@32 (the probability of getting the right answer in 32 tries), the gain reached +13.8 points for MiniCPM.
  • Difficulty Sensitivity: The gains were most pronounced on problems of intermediate difficulty—problems that the baseline model solved only some of the time.
Figure 4: Mean answer length over all evaluation an- swers, per data seed.

Limitations and Future Work

  • Missing Entropy Baseline: While MCPO beats standard GRPO, the paper does not directly compare it against recent entropy-based credit assignment methods. It remains to be seen if margin is strictly superior to entropy in practice.
  • Confounding Length: Because MCPO models produce much longer answers, it's difficult to tell how much of the accuracy gain comes from the "better credit" versus simply "thinking longer." A length-matched baseline would be needed to decouple these effects.
  • Task Scope: The study focused exclusively on competition mathematics. Whether this margin-based credit assignment translates to other reasoning tasks (like coding or symbolic logic) is an open question.

Summary updated

Abstract

In reinforcement learning with verifiable rewards, a language model is rewarded for a correct final answer, and GRPO passes that reward to every token of the solution in equal measure. We instead place the credit where the model hesitated, in a method we call MCPO (Margin-Credit Policy Optimization). We measure hesitation by the top-2 margin $M=p_1-p_2$ of the policy's next-token distribution, map it to a score with two fixed thresholds, and spread each solution's advantage over its tokens so that the mean stays unchanged. The margin is read from a forward pass the trainer already performs, so the method needs no extra model evaluation and no batch statistics. We train MiniCPM5-2B-Base with LoRA for 200 steps on 2,048 DAPO-Math problems and evaluate on six competition-math benchmarks. MCPO beats an unmodified GRPO baseline for all five data seeds, by 6.55 points of macro avg@32 on average (95\% CI $[4.1, 9.0]$), and a Qwen3-4B-Base replication improves for all three seeds ($+4.75$). The gain is largest on problems that GRPO solves only some of the time and grows with $k$ in pass@$k$, reaching 13.8 points at pass@32. MCPO models also write two to three times longer answers, and runs that write longer score higher; our experiments do not separate the two effects.
BibTeX
@misc{liu2026margin,
  author = {Chi Liu and Xin Chen and Yuchuan Weng and Srinivasan Manoharan},
  title = {{Margin-Guided Token Credit for Reinforcement Learning with Verifiable Rewards}},
  year = {2026},
  eprint = {2611.14015},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.14015},
}
Figure