Binary Reward Discretization for Stable Reinforcement Learning

23 concepts17 papers1 proposal

In reinforcement learning (RL), standard intuition suggests that dense, continuous reward signals are superior to sparse, discrete ones. A continuous scalar score—whether measuring semantic similarity, spatial distance, or stylistic alignment—theoretically provides rich gradient information, guiding the agent smoothly toward an optimal policy.

However, optimizing over continuous reward landscapes often introduces severe instability in practice. Continuous reward functions are frequently noisy, highly sensitive to superficial features, and prone to local optima. To counter these vulnerabilities, a growing body of research demonstrates that discretizing rewards into binary signals (such as pass/fail, success/failure, or improvement/degradation) paradoxically yields more stable training trajectories and superior final performance. By stripping away non-informative variance, binary discretization transforms a fragile optimization landscape into a robust, clear supervision signal across domains ranging from large language models to robotics and medical imaging.

Mitigating Noise and Preventing Reward Hacking

Continuous reward models, particularly those derived from neural network discriminators or large language model (LLM) judges, suffer from high variance and poor calibration. When an agent optimizes against a continuous probability or similarity score, it quickly learns to exploit the flaws in the reward model—a phenomenon known as reward hacking. The agent maximizes the score by focusing on superficial traits rather than the underlying objective.

Tong Chen et al. demonstrate this vulnerability in Train for Truth, Keep the Skills: Binary Retrieval-Augmented Reward Mitigates Hallucinations. They show that continuous factuality scores often cause LLMs to optimize for superficial prose styles rather than actual truthfulness, which degrades the model's general utility. By replacing these scores with a strict binary pass/fail signal based on real-world evidence, the authors prevent the model from hacking its way to higher rewards, ensuring positive reinforcement is only granted when factual contradictions are truly eliminated.

Similarly, in multi-style text generation, Karin de Langis et al. convert raw probability scores from style discriminators into binary values in Dynamic Multi-Reward Weighting for Multi-Style Controllable Generation. This binarization mitigates the noise of uncertain discriminators, providing a clear success or failure signal that stabilizes the reinforcement learning process.

This stabilization pattern repeats across various modalities:

Stabilizing Gradients via Relative and Structural Feedback

Beyond noise reduction, continuous rewards can cause optimization trajectories to stall or diverge due to erratic gradient magnitudes. For example, in spatial or sequential environments, as an agent approaches a target, raw geometric metrics can become infinitesimally small or fluctuate wildly, destabilizing policy gradients.

To address this, Fernando Navarro et al. utilize a relative binary reward signal in Deep Reinforcement Learning for Organ Localization in CT. Instead of feeding the raw Intersection over Union (IoU) value to the agent, they provide a binary indicator of whether the latest action improved or degraded the metric relative to the previous state. This relative feedback mechanism simplifies the objective and ensures a stable gradient flow, preventing the reward signal from vanishing near the goal.

A similar mechanism operates in behavioral cognition. Yufan Ji et al. show in Communication Emitter Motion Behavior's Cognition Based on Deep Reinforcement Learning that fixed binary rewards for successful classification provide significantly more stable gradients than rewards tied to the magnitude of state transitions, preventing the agent from getting distracted by irrelevant input variations.

When teaching models to adhere to strict structural formats, dense continuous scoring can also confuse the policy. Ruiyang Zhang et al. leverage binary rewards to enforce concise reasoning styles in SketchThinker-R1: Towards Efficient Sketch-Style Reasoning in Large Multimodal Models, noting that strict binary supervision helps the model navigate structural trade-offs far more reliably than nuanced floating-point rewards.

On a macro level, Sujoy Paul et al. show in Learning from Trajectories via Subgoal Discovery that discrete subgoal-based rewards provide a more robust learning signal than continuous value-based reward shaping, serving as clear checkpoints that prevent agents from getting trapped in local optima caused by noisy value functions.

Framing Reinforcement Learning as Classification

Binarization fundamentally alters the mathematical nature of the policy update. By mapping continuous outcomes to a discrete binary set, the reinforcement learning problem can often be reframed as a robust classification or thresholding task, maximizing the margin between distinct classes of behavior.

In DisCO: Reinforcing Large Reasoning Models with Discriminative Constrained Optimization, Gang Li et al. frame RL as a classification task by maximizing the scoring gap between correct and incorrect responses. This discriminative approach eliminates the difficulty bias inherent in group-relative policy methods, which often underweight extremely easy or difficult samples by focusing heavily on group averages.

Binarization also allows practitioners to explicitly control risk and quality thresholds by converting continuous distributions into clear boundaries. Xinran Wang et al. introduce Quantile-Guided Alignment in Beyond Expectations: Quantile-Guided Alignment for Risk-Calibrated Language Models, where continuous rewards are transformed into binary indicator functions based on whether a sample meets a specific percentile threshold. This mathematical mapping allows the optimization process to isolate and eliminate catastrophic tail failures.

Similarly, in combinatorial optimization, Zeren Huang et al. assign binary labels to branching candidates in Branch Ranking for Efficient Mixed-Integer Programming via Offline Ranking-based Policy Learning based on whether they fall within a top utility percentile, allowing the model to learn cleanly from a discrete set of high-quality actions.

Sample Efficiency and Autonomy from Failure

Finally, binary rewards reduce the cognitive and computational burden of reward engineering, allowing agents to learn effectively from sparse or self-generated feedback.

The foundational utility of this concept was highlighted by Marcin Andrychowicz et al. in Hindsight Experience Replay. They proved that simple binary success/failure signals can outperform meticulously engineered continuous functions when agents re-evaluate past failures as successes under alternative goals. This sparse binary framing dramatically reduces the risk of converging on sub-optimal behaviors that exploit flaws in reward shaping.

This principle extends directly to autonomous agents and test-time adaptation:

By flattening the spurious hills and valleys of continuous reward landscapes, binary discretization forces the policy gradient to focus purely on the boundary between success and failure, providing a robust path toward stable convergence.

Key Foundational Work

Road Tracking Using Deep Reinforcement Learning for Self-driving Car Applications (2019) — contributed 2 concept elements to this direction.
Hindsight Experience Replay (2017) — contributed 1 concept elements to this direction.
Learning from Trajectories via Subgoal Discovery (2019) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_binary_reward_discretization_05dfdfd4,
  title = {binary reward discretization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/binary-reward-discretization/rdr_05dfdfd4070948638d07306782ea66c6},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_binary_reward_discretization_05dfdfd4_v1,
  title = {binary reward discretization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/binary-reward-discretization/rdr_05dfdfd4070948638d07306782ea66c6/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake