TL;DR
Current benchmarks for evaluating mathematical reward models—notably the math subset of RewardBench—suffer from severe representation discrepancies between human and machine solutions and rely on isolated one-to-one comparisons, failing to predict real-world policy improvement. To fix this, the authors introduce REWARDMATH, a benchmark built with standardized step-by-step machine completions and a one-to-many evaluation setup (1 correct vs. 9 diverse incorrect solutions). REWARDMATH strongly correlates with downstream policy optimization (such as Best-of-$N$ and DPO) and reliably flags reward overoptimization, offering a much more trustworthy guide for mathematical RLHF.
What Problem Does This Paper Address?
Reinforcement Learning from Human Feedback (RLHF) and inference-time search (like Best-of-$N$ sampling) rely heavily on reward models (RMs) to score candidate responses. If an RM is inaccurate, policy models can exploit its flaws—a failure mode known as reward hacking or reward overoptimization, where proxy reward climbs while actual output quality degrades.
To test reward models directly, researchers often turn to benchmarks like RewardBench. However, its math split has fundamental flaws:
- Distributional mismatch: Chosen solutions are human-written (which frequently skip intermediate steps), while rejected solutions are machine-generated, verbose chains of thought. RMs often learn to exploit step counts or stylistic differences rather than verifying mathematical reasoning.
- Fragile one-to-one evaluations: Comparing a single chosen answer to a single rejected answer only tests isolated edge cases and fails to capture whether an RM can distinguish correct reasoning from a wide spectrum of logical errors.
Because of this, high performance on RewardBench often fails to translate into better downstream policy models.
Methods
The authors propose a principled design for math RM evaluation and instantiate it in REWARDMATH:
- Harmonizing Representations: The authors replace human-written solutions with detailed, step-by-step machine-generated solutions using GPT-4, verified and corrected via manual inspection to eliminate step-count artifacts.
- One-to-Many Comparisons: Instead of binary pairs, each problem features 1 correct solution alongside 9 distinct incorrect solutions collected from 13 different open- and closed-source LLMs, as well as synthetic step-level perturbations.
- Strict & Rank-Based Scoring: In addition to 1-of-10 classification accuracy (the chosen solution must outscore all 9 rejected solutions; random chance = 10%), they evaluate models using Mean Reciprocal Rank (MRR).
- Mitigating PRM Step-Count Bias: Process Reward Models (PRMs) traditionally aggregate step scores via a cumulative product (
prod), which systematically penalizes longer reasoning chains. The authors propose using the geometric mean (geo_mean), neutralizing step-length bias.
Datasets
- REWARDMATH: Built on MATH500 (the standard 500-problem MATH test subset that avoids PRM800K training overlaps). After filtering 17 trivial problems where almost all models succeeded or parsers failed, the final benchmark contains 483 problems, each with 1 chosen and 9 rejected reasoning trajectories.
- Evaluation Datasets: For evaluating policy models aligned by the RMs, the authors use MATH500 (in-distribution), Gaokao-math, SAT-math, and GSM8K (out-of-distribution).
- Training Sets: Synthetic preference datasets (65K pairs) derived from MetaMATH were used to study reward model scaling and overoptimization behavior.
Key Insights and Ideas
- RewardBench has almost zero correlation with policy improvement: A reward model’s score on RewardBench shows virtually no linear relationship with the accuracy gain of a policy optimized via Best-of-$N$ sampling ($r^2 \le 0.128$). In contrast, REWARDMATH shows a strong correlation ($r^2 > 0.8$ on MATH500).
- A window into reward overoptimization: In both synthetic and non-synthetic setups, models scoring poorly on REWARDMATH (e.g.,
Oasst-rm-2.1-pythia-1.4b) suffer from rapid reward collapse during policy optimization as KL divergence increases. High REWARDMATH performers maintain gold reward gains much deeper into optimization. - The PRM product trap: PRMs often look artificially impressive on RewardBench simply because multiplying step probabilities penalizes the longer rejected solutions. Switching from
prodtogeo_meancauses PRM accuracy on RewardBench to crash (e.g., from 94.4% to 81.4% on Math-Shepherd), exposing vulnerability to step bias. - Direct scoring fails LLM-as-a-judge: Generative judges (like LLaMA-3 or Claude-3) struggle to output calibrated absolute scores on math solutions, often assigning identical scores to correct and incorrect chains. Pairwise comparisons substantially improve their discriminatory ability.
Key Results
- Leaderboard Inversions: Models that top RewardBench struggle on REWARDMATH. For example,
Oasst-rm-2.1-pythia-1.4bachieves 95.1% accuracy on RewardBench but drops to 7.04% accuracy on REWARDMATH (below random chance of 10%). Conversely,Internlm2-7b-rewardachieves a solid 37.27% accuracy and 55.17 MRR on REWARDMATH. - Downstream Correlation: Spearman correlation between RM benchmark performance and Best-of-256 policy performance reaches 0.896–0.902 on MATH500 for REWARDMATH, compared to 0.187–0.369 for RewardBench. Similar improvements appear on Direct Preference Optimization (DPO) ($0.725$ vs. $0.156$).
Limitations and Future Work
- Domain Specificity: The current benchmark focuses strictly on mathematical reasoning, where ground-truth correctness is unambiguous. Extending one-to-many benchmarking to open-ended conversational chat or creative tasks remains an open challenge.
- One-to-Many vs. Many-to-Many: To conserve compute and human verification effort, the authors used 1 chosen solution per problem. In ideal settings, evaluating $m$ correct solutions against $n$ incorrect solutions would offer even greater statistical robustness.
- Fixed Sample Size: The number of negative completions was fixed at $n=9$; the authors did not systematically chart the Pareto curve between inference cost and evaluation reliability across varying $n$.
- RL Stability: Due to base-model distribution mismatch, PPO experiments could not be reliably run across all diverse external RMs, restricting empirical overoptimization testing of arbitrary checkpoints to Best-of-$N$ sampling and synthetic settings.
Summary updated