Penalize, Never Praise: Asymmetric Use of Unlabeled Prompts in Reasoning RL

Reinforcement learning with verifiable rewards trains a language model by sampling many solutions to a problem and rewarding the ones an automatic checker confirms as correct. It works well, but it needs a known answer for every training problem. The usual workaround for unlabeled problems is to guess the answer. The model's most common answer across its own samples is treated as correct, and samples that agree with it are rewarded. That guess is exactly what goes wrong. When the model is confidently wrong, it rewards itself for being wrong, and training locks the error in.

We propose using unlabeled problems only to push probability away from answers that are probably wrong, and never to reward answers that are only probably right. Positive reward comes solely from a small set of problems with checked answers. The bet rests on a specific finding: in self-rewarding RL, rewarding wrong answers is what causes collapse, while failing to reward right answers mostly slows learning down (Breaking the Self-Confirming Loop). If that asymmetry holds, a one-sided signal from unlabeled data should capture most of the benefit of pseudo-labeling and avoid its main failure mode, without TraPO's trajectory-matching filter or RLER's ensemble of models.

Core Claim

In semi-supervised reasoning RL, unlabeled prompts help most when their contribution is restricted to negative, unnormalized updates on rollouts flagged as likely wrong. Group-relative normalization quietly turns any negative signal into a positive one, so it must be removed for the unlabeled prompts. Under equal rollout budgets, this "negative-only" variant should match or beat majority-vote pseudo-rewarding and TraPO's filtered pseudo-rewards. It should also stay stable in the regime where those methods fail: problems on which the model's consensus answer is wrong.

Where This Sits

Three recent methods construct positive pseudo-rewards for unlabeled reasoning prompts.

TraPO rewards majority-vote agreement on unlabeled prompts. It keeps only those prompts whose pass-rate history across training epochs resembles that of labeled problems, which it measures by cosine similarity.

RoiRL weights an offline likelihood objective by majority-vote rewards. Its authors concede it cannot recover when the model puts less than half its probability on the correct answer.

RLER breaks the link between the policy and its own scoring with an ensemble of models, soft rewards, and filtering of rollouts the models disagree on.

All three try to make the positive signal trustworthy. We remove the positive signal from unlabeled data entirely. RLER's diagnosis motivates this design, and it also constrains it. RLER found that over-reward drives collapse, so our method must not reintroduce over-reward through the back door.

The design borrows its logic from offline control. There, Unlabeled Data Sharing assigns zero reward to unlabeled transitions instead of trusting a learned reward predictor. That choice reached about 82% success on AntMaze, where supervised reward prediction reached 0%. The same paper warns that zero reward hurts when the unlabeled data contains good solutions.

The analogy does not transfer literally. Under group-relative policy optimization (GRPO), which scores each sample relative to the other samples drawn for the same prompt, a group whose rewards are all zero produces no gradient at all. Literal zero-reward sharing therefore collapses to labeled-only training. Our negative-only update is the closest working analogue. It gives a non-zero signal only on suspected failures and holds the majority at zero.

Spurious Rewards constrains how we evaluate. Its title already signals that training signals in this setting can mislead. Because the file gives no specific number from that paper, we treat it as a warning and build matched random-label controls into every comparison.

Research Program

Training mixes a labeled pool with an unlabeled pool. Labeled prompts receive standard GRPO with a checked 0/1 reward.

For an unlabeled prompt $x$ with $G$ rollouts $y_1,\dots,y_G$, each rollout receives a penalty weight $w_i \in \{0,1\}$. The policy update for that prompt is

$$\nabla \mathcal{L}_{\text{unl}}(x) = -\lambda \sum_{i=1}^{G} w_i \, \nabla \log \pi_\theta(y_i \mid x),$$

where $\pi_\theta$ is the policy and $\lambda$ scales the unlabeled term. Here $\mathcal{L}_{\text{unl}}$ is a loss being minimized, so stepping along $-\nabla\mathcal{L}_{\text{unl}}$ lowers the log-probability of each flagged rollout. There is no group mean subtraction, so unflagged rollouts receive exactly zero advantage rather than an implicit bonus.

Penalties come from two tiers:

  1. Certifiably wrong. A rollout is flagged if it has no parseable final answer, is truncated, or gives an answer outside the valid domain (for example, a non-integer for AIME).
  2. Minority under strong consensus. A rollout is flagged if its answer is held by at most a fraction $\alpha$ of the group, and only when the top answer holds at least a fraction $\beta$.

A false flag at either tier is a false negative, never a false positive.

There is a real hole in this argument. Lowering minority probability shifts probability onto other answers, including a wrong majority. That is an implicit reward. The research question is whether this indirect effect is small relative to the direct over-reward produced by majority-vote reward.

We will measure it directly, borrowing RLER's diagnostics. The labeled diagnostic set holds 500 labeled prompts kept out of training. On it we score the would-be unlabeled signal against ground truth, which gives RLER's reward-noise and over-/under-reward skew metrics. We also track how the probability of wrong-majority answers drifts over training. The working hypothesis is that this drift stays near zero under negative-only training and grows under majority-vote reward.

Baselines

The naive reference is labeled-only GRPO on the labeled pool. Because all-zero groups give no gradient under GRPO, this is also the literal transfer of zero-reward sharing. Any method must beat it, or the unlabeled data is useless.

The common practical alternative is majority-vote GRPO on unlabeled prompts plus checked rewards on labeled ones. This is TTRL-style training with an anchor set. RoiRL reports that TTRL alone reaches 29.8% on MATH500 with Qwen2.5-Math-1.5B, whose base score is 36.0% (One Training Example).

The strongest existing competitor is TraPO. It reports 42.6% average accuracy with 1K labeled and 3K unlabeled samples, against 38.3% for the best fully unsupervised method trained on 45K samples. We reproduce its OpenR1-Math-220k setup and benchmark suite exactly, and we report our reproduced number alongside the published one. Beating TraPO means a higher average on the same suite at the same labeled/unlabeled split, rollout count, and step count, across three seeds.

RLER is run at its recommended ensemble size of two. It reports results within 3.6% of oracle RLVR, but it costs roughly twice the compute. We compare it at matched compute (fewer steps) and at its native budget, and we report both. An oracle run with checked rewards on all prompts gives the upper bound.

Ablations isolate the key ingredients:

  1. Symmetric variant. The same flags plus +1 for the majority answer tests whether removing praise is what helps.
  2. Normalized variant. Group normalization applied to the negative-only signal tests the claim that normalization leaks positive reward.
  3. Single-tier variants. Each penalty tier is used alone.
  4. Random-penalty control. Penalties are assigned to a random subset of the same size, to rule out spurious-reward effects.

Every arm uses the same rollouts per prompt, the same steps, and a matched tuning budget of eight configurations each.

Why not just…?

Why not just use TraPO's filter with majority-vote rewards? TraPO is the main threat on easy and mid-difficulty problems, where consensus is usually right. There it may win, because positive pseudo-rewards carry more information than penalties alone. Its weakness is the subset of problems where the model's consensus is wrong, since it then tracks a false consensus. We therefore report results split by base-model pass rate, and we accept that negative-only training may only win on the hard split. If it loses everywhere, the claim is false.

Why not just raise the confidence threshold and reward only near-unanimous majorities? High-agreement wrong answers are precisely the confidently-wrong case RLER identifies as the source of the self-confirming loop. A threshold filters out noise, but it cannot catch that case. We run this variant at three thresholds; it makes the case-specific weakness measurable.

Why not just skip unlabeled data, since one example reaches 73.6% on MATH500? One-shot RLVR achieved that on Qwen2.5-Math-1.5B (One Training Example). That model family is known to respond to weak or even spurious signals, and the one-shot result required very long training. We include a one-shot arm at matched compute and repeat key comparisons on Llama-3.2-3B-Instruct, a model also used by RoiRL. Unlabeled data has to show its value on a model where cheap signals help less.

Proposed Experiments

Main comparison. Training uses OpenR1-Math-220k at 1K/3K and 4K/12K labeled/unlabeled splits. Evaluation covers MATH500, AMC, AIME24/25, GPQA-Diamond, and MMLU-Pro, with greedy accuracy and maj@16.

Hard regime. A second experiment filters the unlabeled pool to prompts where the base model's majority answer is wrong. On this pool, majority-vote and TraPO pseudo-rewards should degrade, and negative-only should not.

Scaling. We grow the unlabeled pool from 3K to 100K at a fixed labeled set. RLER reports stable scaling up to 1M samples, so the curve must at least not bend downward.

Decisive checks. Three outcomes would settle the claim:

  1. The normalized-negative ablation underperforms the unnormalized one, confirming that normalization leaks positive reward.
  2. Wrong-majority probability drift on the labeled diagnostic set is measurably lower under negative-only training than under majority-vote reward.
  3. The random-penalty control gives no gain over labeled-only training.

If the drift is comparable across methods, the asymmetry argument fails. In that case the indirect shift of probability onto a wrong majority is doing the same damage as direct over-reward, and the idea should be abandoned.