When a language model is trained with reinforcement learning on open-ended tasks such as clinical answers or research synthesis, there is no program that can check the answer. Instead, a second language model grades each response against a checklist, called a rubric. Rubric judges make errors in a known direction: they often mark a criterion as satisfied when it is absent, and they rarely make the opposite mistake. This means the judge's "no" can be trusted more than its "yes". Yet current training methods apply both kinds of signal immediately and with equal weight. The trained model then finds responses that collect false "yes" verdicts, and quality measured by a stronger, more trusted gold judge first rises and then collapses.
We propose to treat the two kinds of judge verdict differently, borrowing a mechanism from hippocampal plasticity. Penalties from the cheap judge are applied right away. Rewards are held as provisional tags and become reinforcement only after a delayed, harder-to-fool confirmation check. Rewards that are never confirmed turn into a mild penalty. The two-sentence version: trust the judge's negative verdicts immediately, and treat its positive verdicts as claims that must be confirmed before the model is reinforced for them. If the gold-quality collapse comes from one-sided judge errors, deferring and gating only the positive channel should remove most of it at a fraction of the cost of running the trusted check on everything.
Core Claim
Reward hacking under rubric judges is mainly a positive-channel problem, and it calls for a valence-asymmetric update rule rather than for less reward overall. Two strands of evidence support this.
The first comes from the language-model literature. Rethinking Sample Polarity in RLVR separated learning from rewarded and penalized samples in math training. Positive-only updates sharpen the model into shortcut heuristics and reward hacking, while negative-only updates keep exploration alive but degrade text into gibberish. Rubric Dropout reports that rubric judges show "overclaim bias", and that training against static rubrics produces the classic over-optimization curve: training-judge score near ceiling while gold score collapses.
The second comes from neuroscience. Sequential neuromodulation of Hebbian plasticity showed in hippocampal slices that acetylcholine present during activity pushes synapses toward weakening. Dopamine arriving minutes later converts that weakening into strengthening. In navigation models, this yields fast unlearning of stale paths and selective locking-in of rewarded ones. Dopamine and serotonin interplay for valence-based spatial learning found that applying potentiation and depression according to the final trial outcome keeps weights stable. By contrast, letting the two signals compete within an episode can freeze or cycle at high learning rates.
We use these results only as design hypotheses. We make no claim of biological fidelity, and the slice result's in vivo relevance is itself open. Taken together, they suggest three properties for the update rule: a default of weakening, a delayed and selective conversion to strengthening, and decisions made per outcome.
Mechanism
For each prompt, we sample a group of responses and score them with the cheap rubric judge, as in GRPO. GRPO is group-relative policy optimization, which centers each response's reward on its group mean. Responses with negative advantage are updated immediately, exactly as in GRPO. This is the trusted, pruning channel.
Responses with positive advantage are not applied. Each is stored with a tag: its token log-probabilities under the policy that generated it, and the list of criteria the judge claimed were met. A confirmation step then runs on the tagged responses, up to a fixed fraction of them. For each claimed criterion, the judge must quote a supporting span from the response. A string match checks programmatically that the span exists. A second judge, from a different model family, sees only the criterion and the quoted span and decides whether the span really satisfies the criterion. This makes the positive claim partly verifiable, because a judge cannot point to text that is not there.
The confirmed fraction $c_i \in [0,1]$ of response $i$'s claimed criteria then sets the delayed update for that response:
$$ \Delta\theta \propto \sum_i \rho_i \,\big(c_i - \lambda(1-c_i)\big)\, A_i^{+}\, \nabla_\theta \log \pi_\theta(y_i \mid x_i) $$
The symbols are defined as follows. $A_i^{+}$ is the stored positive advantage. $\lambda \ge 0$ is the strength of the default weakening applied to unconfirmed claims. $\rho_i$ is a clipped importance ratio between the current policy and the policy that generated $y_i$, which corrects for the few optimizer steps of delay. $\pi_\theta(y_i \mid x_i)$ is the policy's probability of the response.
Tags that are not confirmed within a fixed window of steps simply expire. Expiring tags give the slice experiments' "depression by default" its analog: a reward is never credited on the cheap judge's word alone.
Research Program
The work has three parts.
A controlled hacking testbed. We train on MATH with an LLM rubric judge, while exact-match answers serve as a hidden gold signal. We inject a tunable overclaim rate into the judge so that the over-optimization curve can be measured precisely and its cause is known.
The open-ended setting. We train on RubricHub and RaR-Medicine/RaR-Science, the data used by Rubric Dropout and Rubrics as Rewards. Evaluation is out of domain on ResearchQA with a gold judge, on HealthBench-Hard with physician criteria (HealthBench), and on GPQA-Diamond.
Mechanism analysis. We measure the judge's overclaim rate on policy outputs across training. If the claim is right, the deferred method should keep this rate flat while GRPO drives it up.
Models are Qwen3-4B and Qwen3-8B, matching the Rubric Dropout setup.
Baselines
Naive reference. The do-nothing reference is GRPO against the full static rubric with the cheap judge. Over-optimization is documented for exactly this configuration, so it shows the size of the problem.
Strongest existing method. The strongest competitor is Rubric Dropout, which randomly masks 30–50% of rubric criteria at each update. It is the only method in the file aimed squarely at rubric hacking. It reports +6 to +7 out-of-domain gold points on ResearchQA with Qwen3-8B and 4B, while keeping about 97% of in-domain training performance. No reported result was found on Lacuna for Rubric Dropout on HealthBench-Hard. We will run Rubric Dropout ourselves at both masking rates, using the same data, models, rollout counts and tuning budget. Beating it means a higher peak and a higher final ResearchQA gold score, with the gap at the final checkpoint larger than seed variance across three seeds.
Rubric reward design. We include Rubrics as Rewards with implicit aggregation. It reports 31.2 on HealthBench, a 31% relative gain over Likert judges, and serves as the reference for how much the rubric reward itself contributes.
Polarity shaping ported to rubrics. We port A3PO from Rethinking Sample Polarity to the rubric setting. On AIME25 it reports Avg@32 of 40.4 versus 35.0 for GRPO on Qwen2.5-7B-Math. This tests whether token-level asymmetric reweighting, without confirmation, already captures the benefit. Rubric Dropout's finding that criteria reweighting can concentrate hacking pressure predicts that it will not.
Ablations of the key ingredient. Three ablations isolate what matters. The first is constant down-weighting of all positive advantages, with no confirmation. The second applies confirmation to all samples, negatives included, which removes the asymmetry. The third uses immediate confirmation with $\lambda = 0$, which removes the default-weakening path.
Fairness rule. Every method is charged the same total number of judge calls. Confirmation calls count against this budget, so the deferred method trains on fewer rollouts per step.
Why not just…?
Why not just use Rubric Dropout? It is the real threat. Its own authors could not tell whether it works by breaking correlations between criteria or by acting as accidental early stopping. If it is the latter, a gated method that never credits unconfirmed claims should keep improving after the point where dropout's gold curve flattens. We will plot gold score against step for both methods to settle this. The two methods are also compatible, and we will test the combination. If dropout alone matches us at equal judge cost, we will report that outcome.
Why not just use the trusted confirmation check as the reward for everything? The quote-and-entail check costs several judge calls per criterion. Under a fixed budget, applying it everywhere means far fewer rollouts. The asymmetry hypothesis says this spending is wasted on negative verdicts, which the cheap judge already gets mostly right. The "confirm everything" ablation at equal cost tests this directly. If that ablation wins, the asymmetry claim is false.
Why not just shrink positive advantages by a constant? This is the cheapest version of "distrust yes". Polarity results suggest it would help somewhat. But a constant factor shrinks genuine and hacked positives equally. Hacked positives would still accumulate, only more slowly, so the collapse would be delayed rather than avoided. The constant down-weighting ablation, swept over its factor, decides the question. If it reaches our peak and final gold score, the confirmation step is unnecessary and the idea collapses to a one-line change.
Why not just raise the KL penalty? A KL penalty keeps the trained model close to the original model's output distribution. Anchored Policy Optimization shows that a global KL penalty creates a conflict between sharpening and preserving entropy. More fundamentally, it limits how far the policy moves without regard to which direction is being hacked. We include GRPO with tuned KL as part of the naive reference.
Decisive Checks
The first decisive test is the synthetic overclaim testbed. Here gold accuracy is exact, so we can sweep the injected overclaim rate. The prediction is specific: gold accuracy for GRPO and dropout should degrade as the overclaim rate rises, while the deferred method should stay nearly flat. If the deferred method also degrades with the overclaim rate, the mechanism is not doing what we claim.
The second test is in the open-ended setting. On ResearchQA and HealthBench-Hard, the method must exceed Rubric Dropout's out-of-domain gold score at equal judge budget. Its gap between training-judge score and gold score must stay bounded through training. GPQA-Diamond accuracy must not fall below that of the RaR-trained model.
The third test checks whether the delay itself matters, which is the specific contribution of the biology. We vary the confirmation window from zero to several steps. If gains appear only at zero delay, the sequencing story is decorative and the result is simply "audit positives". That would still be useful, but it is a smaller claim, and we will report it as such.
References
- Rethinking Sample Polarity in Reinforcement Learning with Verifiable Rewards
- Rubric Dropout: A Simple Way to Mitigate Reward Hacking in Rubric-as-Reward RL
- Sequential neuromodulation of Hebbian plasticity offers mechanism for effective reward-based navigation
- Dopamine and serotonin interplay for valence-based spatial learning
- Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains
- HealthBench: Evaluating Large Language Models Towards Improved Human Health
- Anchored Policy Optimization: Mitigating Exploration Collapse via Support-Constrained Rectification