When a language model is trained with reinforcement learning to solve math problems, the cheapest reward is an outcome check: did the final answer match? That reward cannot see the reasoning. It pays equally for a clean derivation and for a chain of thought (the step-by-step text a model writes before answering) that is bloated, circular, or simply decorative, meaning unrelated to how the answer was actually reached. The alternative is to have a large model act as a judge and grade every reasoning trace, and that costs far more. Practitioners compromise with a cascade. A cheap screener flags suspicious-looking traces, and only those are sent to the expensive judge. This proposal argues that the cascade is itself a mechanism in the economist's sense. The policy being trained is a self-interested agent, and the screener's rule is observable in its consequences. The policy will learn to look unflagged instead of learning to reason better.
The core claim is that inspection-game reasoning from mechanism design predicts the right fix. When the audit probability depends on features the policy controls, the audited penalty must be reweighted by the inverse of that probability. The audit probability also needs a floor. This makes the expected reward independent of whether a trace gets audited. The reweighting has a cost, because variance grows as audits become rarer. That yields a concrete three-way trade-off among verification compute, resistance to gaming, and reward signal quality. We propose to measure this trade-off empirically on reasoning RL. We also propose a budget-constrained audit rule that sits at a better point on it than either outcome-only rewards, full judging, or uniform random spot checks.
Core Claim
Let $x$ be a sampled reasoning trace. Let $o(x)$ be its outcome reward. Let $j(x)$ be the expensive process judge's score of the trace, and let $b$ be a running baseline for that score. Let $p(x)$ be the probability that the trace is sent to the judge, as set by a cheap screener. Let $A \sim \text{Bernoulli}(p(x))$ indicate whether it was. The usual cascade pays $o(x) + A\,(j(x)-b)$. In expectation that equals $o(x) + p(x)\,(j(x)-b)$. For any trace the judge would dislike ($j<b$), the policy therefore gains simply by lowering $p(x)$, that is, by evading the screener. The audited reward we study is
$$R(x) = o(x) + \frac{A}{\max(p(x),\epsilon)}\,\bigl(j(x)-b\bigr),$$
where $\epsilon$ is a floor on the audit probability. For every trace with $p(x)\ge\epsilon$, the expected payment equals the full-audit reward $o(x)+j(x)-b$. Evading the screener then earns nothing. The added variance scales like $1/\max(p(x),\epsilon)$, and the average audit rate is held to a compute budget $q$. Under that budget, the variance-minimizing choice sets $p(x)$ roughly proportional to the screener's estimate of $|j(x)-b|$. This follows the classical optimal-allocation rule from stratified sampling, which puts samples where the quantity being estimated is most variable.
Nothing here is deep theory. The contribution is empirical. We will show that cascaded judging, used as a training signal, is actually exploited by the policy, and we will quantify how much. We will test whether priced audits recover full-judge behavior at a fraction of the compute. We will also test whether the audit design, and not just the budget, matters.
Relation to Prior Work
The mechanism-design ingredient comes from A Theory of Data Acquisition and Pricing at Scale. That paper shows that replacing expensive data valuations with cheap non-monotone proxies invites manipulation by strategic sellers, a "Compute-Truthfulness-Utility Trilemma." Its remedy includes randomized self-resampling that yields truthful payments in expectation from a single evaluation call. It also proves a back-loaded compute schedule, in which cheap evaluation comes early and expensive verification is reserved for late in training. We borrow both the randomized-payment idea and the schedule. We move them from sellers with private costs to a policy whose "private information" is its own reasoning process. We do not inherit the paper's convexity guarantees, so our claims are purely empirical.
The closest judge-side work optimizes judges rather than the audit rule that deploys them. Characterizing, Evaluating, and Optimizing Complex Reasoning trains a Thinking Reward Model (TRM) on traces that all reached the correct answer, which forces it to grade structure rather than outcome. Used as an auxiliary reward on every rollout in GRPO (group relative policy optimization, a common RL method for language models), TRM gives up to a 3.9% gain. J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learning trains judges that reason at length before scoring. Its authors note both the large inference cost and the absence of any guarantee against evaluated models exploiting the judge. Both papers supply our expensive judge. Neither asks what happens when that judge is applied to only a subset of rollouts chosen by a rule the policy can influence.
Truthfulness Despite Weak Supervision is the one paper in the file that brings incentive compatibility into LLM evaluation. It uses peer prediction, which rewards a report by how much it helps a weak model predict a peer's answer. It scores whole outputs and assumes no coordination among participants. We address a different incentive problem: the trained policy exploiting the verifier's sampling rule.
The motivating evidence that outcome rewards let bad traces through is strong. Corruption probes find only 56.1% pooled faithfulness across eleven frontier models, and models catch injected errors in their own traces 1.5% of the time (Auditing Chain-of-Thought Faithfulness). Medical chains of thought decouple from answers at a 72.9% rate (Right Diagnoses, Decorative Reasoning). Two models with near-identical answer susceptibility (1.3% vs 1.2%) acknowledge injected bias at 13.0% vs 75.0% (Beyond Accuracy).
Two constraining results shape the design. Can A Base LLM Verify Its Own Math? shows that verifiers anchor on the final answer string. The process judge must therefore be validated on answer-relabeled corruptions, or it will just re-pay the outcome. Judging with Many Minds shows that sequential debate amplifies bias, which is why we use a single independent audit judge rather than debating judges.
Research Program
We will first build the judges. The expensive auditor is a TRM-style trace judge. Before any RL, we check it on answer-relabeled corruption, meaning traces where an intermediate error and the final answer are changed together so that they agree. The auditor passes only if it penalizes these traces rather than rewarding the self-consistent wrong answer.
We will then build several cheap screeners, each exposing a feature the policy can plausibly manipulate. One is a small judge model. One is a trace-length and hesitation-marker heuristic. One is sample disagreement within a GRPO group.
Next we train policies with GRPO on MATH and GSM8K training problems under matched audit budgets $q$ of 5%, 15%, and 30%. We compare four ways of choosing which rollouts get audited. The first is naive cascades, which audit flagged traces and apply no reweighting. The second is uniform random audits with reweighting. The third is our priced targeted audits, which use the reweighting and a floor and allocate audits in proportion to screener-estimated judge deviation. The fourth adds a back-loaded variant that ramps $q$ upward over training, following the data-acquisition paper's schedule.
The central diagnostic is screener drift. We measure how the fraction of rollouts the screener would flag changes over training, while tracking whether the full judge's score of those unflagged traces rises or falls. Gaming shows up as fewer traces flagged while the auditor, run on everything offline, sees no improvement or a decline.
Baselines
The naive reference point is outcome-only GRPO, rewarded solely by final-answer correctness. It is the default the proposal must beat on trace quality without losing accuracy. It is run on the same problems, seeds, and rollout counts.
The strongest existing method is full-audit training, with the TRM-style judge applied to every rollout as an auxiliary reward. This is the setting in which Characterizing, Evaluating, and Optimizing Complex Reasoning reports up to a 3.9% gain from GRPO with TRM rewards. The same paper reports a 19.3% AIME24 gain for TRM Best-of-N selection. This baseline is the ceiling on signal quality at maximal compute. We run it with identical policy, data, and hyperparameter search budget, and count judge tokens as compute. Priced audits "beat" it if they match its AIME24 accuracy and offline judge scores within seed noise at 15% of its judge compute or less. Matching at 30% would still be a useful but weaker result.
The naive cascade is the practice we claim is broken. It is given the same screener and budget as priced audits and differs only in lacking reweighting and a floor. Beating it means measurably less screener drift together with higher offline judge scores and faithfulness.
The key ablation is uniform random auditing with reweighting at the same budget. It isolates whether targeting adds anything beyond unbiased subsampling. A second ablation removes the floor $\epsilon$, to show the variance cost the floor controls.
For judge quality, the file reports J1-Qwen-32B at 93.6 on RewardBench and 76.8% on PPE Correctness (J1). We will report our auditor's RewardBench score alongside these to show it is a competent judge. For evaluating RL under partial audits, we found no reported result on Lacuna for cascaded or subsampled judge rewards.
Why not just…?
Why not just use outcome rewards and accept the traces? Because the evidence says outcome rewards leave trace quality unconstrained. They produce no gradient once the answer is correct, as the TRM paper's plateau argument shows. They coexist with 56.1% faithfulness and a 72.9% decoupling rate in the audits cited above. Our outcome-only arm tests this directly with the corruption probe and the bias-acknowledgment rubric. If outcome-only policies score as well there as audited ones, the proposal's premise fails. We will report that honestly.
Why not just judge every rollout? If compute is no object, this is the right answer and the proposal has no advantage. The question is only whether a fraction of that compute buys the same policy. J1's own authors flag hundreds of judge tokens per evaluation as a real overhead, and judge cost grows with rollout count. The full-audit arm sets the target. The budget sweep shows where, if anywhere, priced audits fall below it.
Why not just audit uniformly at random? This is the most serious threat. Reweighted uniform sampling is already unbiased and cannot be gamed. The only possible advantage of targeting is lower variance at a fixed budget. If the screener's estimate of judge deviation is uninformative, targeting buys nothing, and the practical recommendation becomes "randomize and reweight." Even in that case, the naive-cascade finding stands as the main result. The uniform-audit ablation settles which of these holds.
Why not just rerank samples with the judge at test time? Best-of-N reranking already gives large AIME24 gains. But it improves which answer is selected, not how the policy reasons. It also pays judge cost on every query at deployment. We therefore report it as a separate reference line rather than a competitor for training-time faithfulness.
Proposed Experiments
All policies share one base model, the MATH and GSM8K training split used for GRPO, and equal rollout counts. We evaluate on AIME24 accuracy and on offline full-auditor scores over held-out traces. We also run the corruption-probe protocol of Auditing Chain-of-Thought Faithfulness, against its reported 56.1% pooled faithfulness, with the protocol expanded beyond its 30 problems. We add GSM8K bias acknowledgment under the B4–B6 injections of Beyond Accuracy. Finally, we record trace length, to catch verbosity-driven judge exploitation, and judge compute.
The first decisive check is whether naive cascades show screener drift. That means a falling flagged fraction without a matching rise in offline judge score. It should appear for each screener feature, most strongly for the length heuristic. If no screener is evaded after a full training run, the gaming hypothesis is false and the paper becomes a compute-saving study.
The second check is whether priced audits at $q=15\%$ match full-audit AIME24 accuracy and offline judge scores across three seeds, while naive cascades at the same budget do not.
The third check is whether targeted priced audits beat uniform reweighted audits at equal budget in reward-gradient variance and final judge score. The fourth is whether the back-loaded schedule outperforms a constant $q$ with the same total compute, as the data-acquisition analysis predicts.
Negative outcomes on the third and fourth checks narrow the contribution rather than sink it. Negative outcomes on the first and second do sink it.
References
- A Theory of Data Acquisition and Pricing at Scale
- Characterizing, Evaluating, and Optimizing Complex Reasoning
- J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learning
- Truthfulness Despite Weak Supervision: Evaluating and Training LLMs Using Peer Prediction
- Auditing Chain-of-Thought Faithfulness for Trustworthy AI: A Reproducible Corruption-Probe Protocol Across Eleven Frontier LLMs
- Right Diagnoses, Decorative Reasoning:A Perturbation Audit of Medical Chain-of-Thought
- Beyond Accuracy: Measuring Bias Acknowledgment in Chain-of-Thought Reasoning for Responsible AI Evaluation
- Can A Base LLM Verify Its Own Math? A Pre-Registered Discriminator-Generator Asymmetry Test On AIME And Four Corrective Probes
- Judging with Many Minds: Do More Perspectives Mean Less Prejudice? On Bias Amplification and Resistance in Multi-Agent Based LLM-as-Judge