Introduction: The Challenge of Self-Rewarding RL
Reinforcement Learning with Verifiable Rewards (RLVR) is the gold standard for training reasoning models, providing clear "correct/incorrect" signals. However, verifiable data is often scarce. An alternative is Reinforcement Learning with Intrinsic Rewards (RLIR), or self-rewarding RL, where a model evaluates its own outputs.
Self-rewarding RL allows models to improve without human labels, but it often fails because models become "overconfident" in their own mistakes. This creates a "policy-reward coupling" where errors are reinforced, leading to training instability and a performance ceiling significantly lower than what is achievable with ground-truth labels.
Methodology: Breaking the Feedback Loop with RLER
To address these issues, the authors propose RLER (Reinforcement Learning with Ensembled Rewards). This framework is designed to decouple the policy from the reward signal through three main components:
- Ensemble Self-Rewarding: Instead of a single model judging itself, an ensemble of $K$ models generates rollouts and calculates a unified reward. Averaging across diverse models reduces the likelihood that any single model's specific bias will dominate the reward signal.
- Adaptive Soft-reward Interpolation: The system dynamically mixes "hard" binary rewards with "soft" probability-based rewards. A confidence-based gate ($\alpha$) determines the weight: high-confidence ensemble agreements receive harder rewards, while disagreements are treated with more nuance to maintain robustness.
- Confidence-disagreement Balanced Rollout Selection: To prevent "over-rewarding" (False Positives), the system filters rollouts. It specifically down-weights high-confidence errors while retaining scarce correct samples from the "tail" of the distribution.
Finally, the ensemble is consolidated back into a single, high-performing model using Model Merging (Ties-Merging) for practical deployment.
Formalizing and Analyzing System Bias
The study formalizes system bias into three measurable metrics:
- $\rho_{noise}$: The overall magnitude of reward noise.
- $\rho_{selfbias}$: The strength of the coupling between the policy and the reward.
- $\rho_{symbias}$: The imbalance between over-rewarding (False Positives) and under-rewarding (False Negatives).
A key finding is that over-rewarding is significantly more detrimental than under-rewarding. Rewarding an incorrect answer introduces a "near-orthogonal gradient bias" that can cause training to collapse. High self-bias acts as an amplifier: it accelerates convergence when the model is correct but aggressively locks in incorrect reasoning patterns when the model is wrong.
Experimental Results
The researchers evaluated RLER on synthetic arithmetic datasets and the DAPO-MATH-17K reasoning dataset, testing against benchmarks like MATH500 and AIME.
- Performance Gains: RLER improved performance by +13.6% over the best existing RLIR baselines.
- Closing the Gap: The framework reached 96.4% of the performance of RLVR (the theoretical upper bound using ground-truth labels).
- Stable Scaling: Unlike standard self-rewarding methods that often plateau or collapse with more unlabeled data, RLER showed stable, continued improvement as data size increased from 8k to 1024k samples.
- Benchmark Success: On the MATH benchmark, RLER achieved 77.5% accuracy, outperforming standard Self-Consistency (75.0%) and Frequency-based (71.7%) methods.
Limitations and Future Outlook
While effective, RLER introduces higher computational overhead due to the ensemble training, though data sharding can mitigate this. The success of the final model also depends on the effectiveness of model merging techniques like Ties-Merging. Future research will explore whether these bias mitigation strategies generalize beyond mathematical reasoning to domains like creative writing or coding.
Summary updated