Cloze Rewards for Stalled GRPO Groups

Reinforcement learning with checkable answers has become the standard way to teach language models to reason, and its most common recipe, Group Relative Policy Optimization (GRPO), works by comparison. For each problem it samples a handful of solutions, marks each right or wrong, and pushes the model toward the solutions that beat the group average. When every sampled solution is wrong, nothing beats the average, so the learning signal is exactly zero. This happens constantly on the hard problems that matter most. We propose a replacement signal for these stalled groups that comes from the model's own pre-training head. After each failed reasoning trace, we append the known correct answer inside a fill-in-the-blank template. We then ask a frozen copy of the base model how likely that answer is. Traces that make the true answer more predictable get a small positive advantage, even though none of them reached it.

The idea borrows its logic from the older line of work on aligning downstream tasks with the pre-training objective. That work showed that scoring labels through the pre-trained language-model head, instead of a freshly initialized classifier, extracts useful signal from very little data. Here the "label" is the reference answer and the "input" is a reasoning trace. The claim is that a model's next-token distribution already contains a graded, if noisy, judgment of how close a wrong trace came. No reward model, extra parameters, or hand-built virtual reward is needed. If the claim holds, hard training problems stop being wasted compute and start contributing gradient from the first step.

Core Claim

Let $q$ be a problem, $o_i$ the $i$-th of $G$ sampled traces, $a^*$ the reference final answer, and $\tau$ a short answer template such as "Therefore, the final answer is \boxed{ }". We define the cloze score of a trace as the length-normalized log-probability that the frozen reference model $p_{\text{ref}}$ assigns to the reference answer when it is placed after the trace:

$$c_i = \frac{1}{|a^*|}\log p_{\text{ref}}\left(a^* \mid q, o_i, \tau\right).$$

Ordinary GRPO computes advantages from binary correctness rewards $r_i \in \{0,1\}$. Our change applies only when the group has collapsed, meaning all $r_i$ are equal. In that case each trace gets the advantage

$$A_i = \lambda\left(c_i - \tfrac{1}{G}\textstyle\sum_j c_j\right),$$

where $\lambda$ is a small fixed scale. Following the analysis in Understanding R1-Zero-Like Training, we deliberately do not divide by the group standard deviation. That paper, which audits GRPO on base models, shows that standard-deviation and length normalization bias the update. With continuous scores inside a collapsed group, dividing by a tiny spread would also inflate noise. Groups with mixed correctness keep the usual binary advantage untouched, so the verifier remains the sole authority whenever it has something to say.

The claim is empirical and falsifiable. On problems where the model initially solves zero of $G$ samples, collapse-gated cloze advantages should turn more of those problems into solved ones by the end of training than any competing treatment of collapsed groups. They should do so without lowering accuracy on problems that were already learnable.

How This Relates to Prior Work

Two strands in the file motivate the gap. The first shows that stalled groups are a major cost. The Advantage Collapse Rate study behind AVSPO (He et al.) measures how much training compute is spent on all-right or all-wrong groups. It patches the problem with heuristic virtual rewards and reports 4–6 point gains over vanilla GRPO on benchmarks including MATH-500. Our proposal targets the same failure but derives the substitute signal from the model's likelihood rather than a hand-designed rule, and AVSPO is the method we must beat. The dynamics study of Ghosh et al. finds that 70–80% of GRPO's reward gains arrive early in the first epoch and then plateau. This is consistent with the model exhausting its learnable problems and stalling on the rest, which is the regime we aim at.

The second strand supplies the method. Cloze reformulation, standardized in OpenPrompt and pushed to few-shot strength by LM-BFF, scores candidate labels through the pre-trained head. Chada et al. apply the same move to extractive question answering and Ding et al. to entity typing, with large few-shot gains. We reuse this scoring device but change what is scored: the conditioning context is a model-generated reasoning trace, and the scored "verbalizer" is a fixed reference answer. The same literature also constrains the design. Cloze scores are known to swing with template phrasing, and Chen et al. show that prompt-based scoring is overconfident and needs calibration. Separately, Understanding R1-Zero-Like Training reports that a standard prompt template reduced Qwen2.5-Math accuracy by roughly 60%. For these reasons we average over several templates and treat template sensitivity as a measured quantity rather than an afterthought.

The closest method in spirit is Advantage Weighted Matching, which reweights the native diffusion pre-training loss by sample advantages and converges 16.54× faster than Flow-GRPO on GenEval. We deliberately do not port AWM's recipe to text. As that paper itself notes, autoregressive policy gradients already are advantage-weighted next-token log-likelihood. An "advantage-weighted pre-training loss" for language models would largely restate GRPO without its clipping. The same reasoning applies to an online version of VAR, which turns alignment into reward-weighted supervised fine-tuning offline. Its exponential weights still vanish when every reward in a group is equal. What pre-training alignment can genuinely add in language models is therefore not a new loss shape but a new source of reward, and that is where we place the contribution.

Two remaining works bear on regularization. Vojnovic & Yun show theoretically that GRPO's stationary policy is mode-seeking. GRPO-SG stabilizes GRPO by weighting tokens by their own probability. Both modify how existing reward is spent, and neither creates signal in a group that has none. They are complementary to our proposal rather than competing with it.

Research Program

The work has three stages.

The first stage is a pre-training-free validity study, which gates everything else. We sample 16 traces per problem from Qwen2.5-Math-7B and a Llama 3B base model on the MATH training split (Hendrycks et al.), which ships reference answers. We keep problems where all 16 traces fail and compute cloze scores under five templates. To establish ground truth for "closer to right", we truncate each failed trace at 50% and 75% of its length and resample 16 continuations from that prefix. The resulting prefix-continuation success rate estimates how salvageable the trace was. The cloze score is useful only if, within a group, it rank-correlates with this success rate better than the naive proxies of trace length and mean token log-probability.

The second stage is training. Collapse-gated cloze advantages are plugged into a GRPO trainer that uses Dr. GRPO's unnormalized loss as the base. Four design choices are studied in isolation. The first is scoring with the frozen reference model versus the current policy; the frozen model is our default, to avoid the policy learning to inflate its own scores. The second is gating: collapsed groups only, versus adding cloze scores as a tiebreaker inside all groups. The third is single template versus template ensemble. The fourth is the scale $\lambda$.

The third stage probes reward hacking. The model never sees $a^*$ during rollout, so it cannot copy it. It could, however, learn to end traces with a spread of candidate numbers that raises the likelihood of whatever the answer happens to be. We will track two quantities over training to detect this: the number of distinct candidate answers mentioned per trace, and the entropy of the reference model's answer distribution after the trace. As a hard mitigation we will test scoring only the final committed line of the trace.

Baselines

The naive reference point is vanilla GRPO as described in DeepSeekMath, where collapsed groups simply contribute zero gradient. It is also the setting behind DeepSeekMath 7B's reported 51.7% on MATH. We run it on the same base models, data, rollout count $G$, and number of optimizer steps as every other arm.

The strongest direct competitor on this question is AVSPO (He et al.), because it attacks precisely the collapsed-group problem with an alternative substitute signal. The file reports its gain only as 4–6 points over vanilla GRPO on benchmarks including MATH-500, so we will reimplement it inside the same trainer. Its hyperparameters get the same tuning budget as ours: an equal number of sweep runs on a held-out MATH slice. Beating it means higher accuracy on the initially unsolved training problems and on MATH-500 and AIME 2024, at matched rollout compute.

The strongest general GRPO variant is Dr. GRPO (Understanding R1-Zero-Like Training), which reports 43.3% on AIME 2024 with Qwen2.5-Math-7B. Our method is built on Dr. GRPO, so this comparison isolates the added cloze signal exactly. We will first reproduce its setup. If our reproduction falls short of 43.3%, we will report both the published and the reproduced numbers and compare against our reproduction. On Knights and Knaves we also include GRPO-SG (Le et al.), which reports 0.63 accuracy versus 0.39 for GRPO on Qwen2.5-3B. We will additionally test whether our method stacks with it.

Three ablations isolate the key ingredient, namely that the signal comes from the pre-training head's judgment of the reference answer. The first replaces $c_i$ with a random score of matched variance, testing whether any nonzero perturbation in collapsed groups helps. The second replaces $c_i$ with the trace's own mean token log-probability, which is pre-training-derived but does not use the reference answer. The third, reference-solution supervised fine-tuning on collapsed problems, is treated in the next section because it is the most serious threat.

Why not just…?

Why not just drop collapsed prompts and resample, as a curriculum would? Filtering saves compute but gives up on exactly the problems beyond the model's current reach, which is where headroom lives. The prediction that separates the two is concrete. A filtering arm, run at matched rollout budget, should match us on already-learnable problems but solve fewer of the initially unsolved training problems and trail on AIME. If it does not trail, the cloze signal is not earning its keep, and we will report that result.

Why not just add a supervised next-token loss on the reference solution whenever a group collapses? This is the classic pre-training-likelihood anchor, similar in spirit to PPO-ptx, and it is a genuine threat, since the reference solution contains more information than a cloze score. The case against it is that it is off-policy. Human-written MATH solutions differ in style and length from the model's own traces, and imitating them can pull the policy away from the exploratory behavior that online RL produces (Understanding R1-Zero-Like Training). The cloze reward, by contrast, only reranks the model's own samples. We run this arm at a tuned mixing weight and compare it on hard-problem solve rate, pass@16, and response-length drift. Pass@16 is included because it measures the diversity that the mode-seeking analysis of Vojnovic & Yun predicts will shrink. If supervised anchoring wins on accuracy, the honest conclusion is that the reference solution should be used directly. The remaining question would then be whether a hybrid helps.

Why not just use in-context experience instead of weight updates? Training-Free GRPO raised a 671B model from 67.9% to 73.3% on AIME 2025 for about $18. However, it relies on a very strong model to articulate lessons from contrasting successful and failed rollouts. In a fully collapsed group there is no successful rollout to contrast against. It also cannot change the weights of the 3B–7B models studied here. We treat it as out of scope for small models rather than as a competitor.

Why not just early-stop vanilla GRPO? Early stopping exploits the plateau found by Ghosh et al., but it leaves the stalled problems stalled. Our method aims to move the plateau itself, which we check by comparing where each training curve saturates.

Proposed Experiments and Decisive Checks

Training uses the 7,500-problem MATH training split. Evaluation covers MATH-500, AIME 2024, GSM8K (Cobbe et al.), and OlympiadBench for in-domain generalization, and Knights and Knaves for a non-arithmetic domain with checkable answers. MMLU-Pro serves as a check that general knowledge is retained. Models are Qwen2.5-Math-7B and a Llama 3B base, with three seeds per arm. Metrics are pass@1 and pass@16, the Advantage Collapse Rate over training, the fraction of initially unsolved training problems solved at the end, mean response length for correct and incorrect traces, and the two hacking diagnostics from stage three.

Three results settle the question. The first is the stage-one validity study. If the within-group rank correlation between cloze score and prefix-continuation success is not clearly above that of the trace-log-probability proxy, we stop, because the signal is not there. The second is the hard-subset comparison. Collapse-gated cloze GRPO must solve more initially unsolved problems than AVSPO, reference-solution fine-tuning, and the random-score ablation at matched compute, while matching Dr. GRPO on the rest. Relative to Dr. GRPO's reported 43.3% on AIME 2024, the meaningful outcome is a gain that exceeds seed variance, not merely parity. The third is template robustness. Accuracy under the template ensemble should vary across held-out templates by less than the method's gain over Dr. GRPO. Otherwise the result reproduces the fragility the cloze-prompting literature warns about rather than overcoming it.