Calibrated Decision Rewards: Training Language Models with a Judge That Knows How Sure It Is

Rudransh Agnihotri, Manasvi KapoorarXiv 2026 · arXiv

TL;DR

Researchers have developed RLCDR (Reinforcement Learning from Calibrated Decision Rewards), a method that turns an LLM into a "decision model" capable of providing trustworthy, calibrated probabilities for its own judgments. By using these calibrated rewards instead of fragile reward models or rigid answer keys, they improved a 1.7B model's math reasoning to match the performance of training with a ground-truth answer key—without ever showing the model the reference answers.

What Problem Does This Paper Address?

Reinforcement Learning (RL) is the engine behind recent leaps in LLM reasoning (like DeepSeek-R1 or OpenAI’s o1), but it’s only as good as its reward signal. Currently, researchers face a frustrating trade-off:

  1. Answer Checkers: These are exact (e.g., checking if a math answer is "42"), but they only work when you have a reference answer key. They provide no signal for open-ended tasks or for "nearly right" failed attempts.
  2. Learned Reward Models (RMs): These can judge anything, but they are notoriously "brittle." They are often overconfident, biased by the order of options, and prone to "reward hacking," where the policy learns to exploit the RM's flaws (like making answers longer to get higher scores) rather than actually getting smarter.

The authors argue that the missing ingredient is calibration: a reward whose numbers actually mean what they say. If a reward model says "0.8," it should be right exactly 80% of the time.

Methods

The authors introduce RLCDR, which converts a 27B parameter LLM into a Decision Model. Unlike standard LLM judges that generate text critiques, this model outputs a probability distribution in a single forward pass.

The pipeline involves three key steps:

  1. Typed Questions: The judge is asked specific multiple-choice questions like "Is the final answer correct?" or "Which step is the first error?"
  2. Debiasing: To fix "position bias" (where models prefer the first option), they run the question with options in different orders and average the results.
  3. Calibration (Platt Scaling): They use a tiny labeled dataset (~100 examples) to fit two parameters that map the model's raw scores to true probabilities. This ensures the reward stays "honest" even as the policy it's training changes.
The RLCDR pipeline: from policy answers to calibrated probability rewards.

These calibrated probabilities are then used as the reward signal for GRPO (Group Relative Policy Optimization), a popular RL algorithm that compares multiple completions for the same prompt.

Datasets

  • Training: The MATH training split (levels 3–5).
  • Evaluation: MATH500 (a standard subset of the MATH test set), AIME 2024/2025, and ProcessBench (for evaluating step-by-step error detection).
  • Calibration: A small set of 1,024 labeled responses from the PPE (Policy Performance Evaluation) dataset.

Key Insights and Ideas

  • Calibration is the "Truth Serum" for RL: A calibrated reward allows the RL objective to act as a principled Bayesian update. It prevents the model from over-correcting on uncertain signals.
  • Reference-Free is Possible: The most striking finding is that a judge that has never seen the answer key can train a model just as effectively as the answer key itself.
  • The "Hard Problem" Signal: When a model fails every single attempt at a hard problem, an answer key provides zero gradient (everything is 0). However, the RLCDR "critic" can recognize that one failed attempt got 80% of the way there, providing a "step-level" reward that helps the model learn from its mistakes.
  • No Reward Hacking: Because the reward is calibrated to true accuracy, the authors found no evidence of the model "gaming" the system (e.g., by increasing response length) to get higher scores.

Key Results

The researchers trained a Qwen3-1.7B model using RLCDR and compared it against several baselines:

  • Matching the Ceiling: RLCDR raised MATH500 accuracy from 72.2% to 76.8%. This statistically matched the performance of training with the actual Answer Key (76.9%).
  • Difficulty Scaling: The gains were most pronounced on the hardest problems (Level 5), jumping from 44.8% to 51.7%.
  • Superior Error Detection: Even without specific training on reasoning steps, the 27B decision model outperformed a dedicated 7B Process Reward Model (PRM) at locating the first error in a solution (ProcessBench F1 of 0.772 vs. 0.722).
  • Efficiency: Because it uses a single forward pass and no generated text, the reward is significantly cheaper to compute than "generative" judges that write long critiques.
Figure 1. The policy improves because of the reward. Qwen3-1.7B trained with GRPO on MATH (training split), changing only the reward. Every evaluation uses reference answers, never the reward model. Left: accuracy after 100 steps, with 95% paired-bootstrap intervals for the gain over the base model. Right: our reward improves the policy at every checkpoint and ends level with the answer key. Random reward does not help. Each evaluated checkpoint carries about ±1.8 points of 95% uncertainty.

Limitations and Future Work

  • Overconfidence on the "Impossible": On problems where the base model fails almost 100% of the time, the judge's calibration starts to slip, becoming overconfident. The authors suggest that recalibrating on a small set of specifically "hard" problems is a necessary next step.
  • Domain Specificity: This study focused heavily on mathematics. While they showed some transfer to coding and science (GPQA), more work is needed to see if these calibrated rewards hold up in subjective domains like creative writing or general chat alignment.
  • Single Seed: The RL results use a single seed. While the effects are large enough to be statistically significant, the authors acknowledge that multi-seed runs are needed for definitive policy-level claims.

Summary updated

Abstract

Reinforcement learning (RL) can only teach a language model what its reward can recognize. Answer checkers are exact but need a reference answer; learned reward models and LLM judges work everywhere, but their scores are not probabilities. We study RLCDR (Reinforcement Learning from Calibrated Decision Rewards). An LLM converted into a decision model answers a typed question about a response ("is the final answer correct?", "which step is the first error?") with a probability distribution, read in one forward pass, debiased over option orders and calibrated with about a hundred labels. Training Qwen3-1.7B with GRPO on MATH, this reference-free reward raises MATH500 accuracy from 72.2% to 76.8% (+4.65 points, p < 1e-4), matching training on the answer key (76.9%), while a random reward gains only +0.65. The gain grows with difficulty, from 44.8% to 51.7% on the hardest level. On the policy's own changing samples the reward stays calibrated: its expected calibration error is half that of a softmax judge on the same 27B backbone (0.074 vs. 0.141), and it shows no sign of reward hacking. Without step training, the same model finds the first wrong step better than a dedicated 7B process reward model (ProcessBench F1 0.772 vs. 0.722, 20/20 splits). We also map where the approach stops being enough: trained only on problems the base model mostly fails, the reference-free reward still improves the policy (+1.9 MATH500 points, p = 0.03) but trails the answer key, because its probabilities become overconfident on that distribution. We state six hypotheses in advance and test each with a control designed to falsify it.
BibTeX
@misc{agnihotri2026calibrated,
  author = {Rudransh Agnihotri and Manasvi Kapoor},
  title = {{Calibrated Decision Rewards: Training Language Models with a Judge That Knows How Sure It Is}},
  year = {2026},
  eprint = {2611.25135},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.25135},
}
Figure