The recent success of models like DeepSeek-R1 has centered on a specific training paradigm: Reinforcement Learning (RL) guided by verifiable, binary rewards. In tasks like mathematics or coding, a model is rewarded if the final answer is correct and penalized if it is not. While effective, this "pass/fail" approach creates a sparse and unforgiving learning signal.
A growing body of research is now focused on moving beyond these binary constraints toward continuous reward signals and advantage shaping. The goal is to provide models with a smoother gradient for learning, allowing them to distinguish between a "near-miss" and a total failure.
The Sparse Signal Problem
In standard RL for reasoning, such as Group Relative Policy Optimization (GRPO), the reward is typically a discrete $\{0, 1\}$. As noted in olmOCR 2: Unit Test Rewards for Document OCR by Jake Poznanski and colleagues, while binary unit tests provide clear signals, they lack the granularity to reward incremental progress.
If a model produces a complex 20-step mathematical proof where 19 steps are perfect but the final calculation is slightly off, a binary reward treats it the same as gibberish. This makes the "training frontier" difficult to navigate, as the model receives no positive reinforcement until it happens to stumble upon the complete, perfect sequence.
Advantage Shaping as a Mathematical Bridge
To solve this, researchers are unifying different optimization objectives under the umbrella of surrogate reward maximization. Christos Thrampoulidis, Sadegh Mahdavi, and Wenlong Deng demonstrate in Advantage Shaping as Surrogate Reward Maximization that direct optimization of metrics like "Pass@K" and "advantage shaping" (adjusting reward scores based on difficulty or baseline) are mathematically equivalent.
This unification suggests a "recipe" for better RL: instead of just rewarding the outcome, we can shape the advantage—the measure of how much better a specific response is compared to the average. By refining how we calculate this advantage, we can inject "partial credit" into the training loop.
From Outcomes to Process Rewards
One of the most promising directions for creating continuous signals is the use of Process Reward Models (PRMs). Rather than waiting for the final answer, PRMs evaluate the "correctness" of each intermediate reasoning step.
- Step-wise Nuance: Junyi Li and Hwee Tou Ng introduce FSPO (Factuality-aware Step-wise Policy Optimization) in Reasoning Models Hallucinate More, which rewards factual accuracy at every step.
- Mining "Gems": Even in a failed trajectory, there are often correct sub-sequences. Yuxiao Ye and Zhaohui Yang propose BCPG-NSA in Unearthing Gems from Stones, a method that segments negative trajectories to find and reward these "gems" of correct reasoning hidden within an overall incorrect response.
- Fuzzy Rewards: In the domain of vision-language models, CrowdVLM-R1 introduces Fuzzy Group Relative Policy Reward (FGRPR). Instead of a binary "correct count," the model is rewarded based on how close its estimate is to the ground truth, allowing it to learn from "nearly correct" guesses.
Efficiency through Regression and Contrast
Moving to continuous rewards often increases computational complexity. To counter this, new frameworks are simplifying the RL pipeline:
- Optimal Advantage Regression ($A^\star$-PO): Kianté Brantley, Wen Sun, and others developed A*-PO, which replaces the complex "critic" model used in PPO with a simple offline regression task. This allows the model to estimate the "optimal" value of a prompt more efficiently.
- Contrastive Learning: Yihong Wu and colleagues argue in It Takes Two: Your GRPO Is Secretly DPO that group-based RL is essentially a form of online contrastive learning. By viewing RL through this lens, they show that even a "minimalist" version with only two rollouts can capture most of the performance gains of much larger groups.
- Single-stream Optimization: Zhongwen Xu and Zihan Ding propose SPO (Single-stream Policy Optimization), which uses a persistent value tracker to eliminate the need for large, computationally expensive groups, further pushing RL toward continuous, global normalization.
The Future: High-Resolution Alignment
The transition from binary to continuous rewards is not just a mathematical convenience; it is a requirement for aligning models on subjective or complex tasks. As Kawin Ethayarajh and the team behind KTO point out, current alignment objectives often lack the mechanisms to process granular data like multi-point ratings.
By developing frameworks that can handle soft estimation—as explored in Athena-PRM by Jiaheng Wei and others—the field is moving toward a future where models are not just told "yes" or "no," but are given the high-resolution feedback necessary to master the nuances of human-like reasoning.