Language models asked to check and fix their own answers often make things worse. They talk themselves out of correct answers, and they reject valid solutions. This proposal argues that part of this failure is an incentive problem, and that economic contract theory gives the right tools to fix it. Contract theory studies how to pay an agent whose effort you cannot see, using only the outcomes you can see. When a model is trained with reinforcement learning to revise its answers, the reward function plays the role of a contract and the trained policy plays the role of an agent responding to it. The trainer cannot observe whether the model genuinely critiqued its answer. It sees only a verdict on the output, and that verdict usually comes from a cheap judge, often another language model, which is badly wrong in exactly the cases that matter.
We propose to design these rewards as contracts. Payment depends on the transition between the initial and the revised answer: whether a correct answer was kept, a wrong one was fixed, a correct one was broken, or the model honestly said the problem has no solution. Verdicts come mostly from a noisy model judge. A small, randomly chosen fraction of episodes is audited by a sound symbolic checker, an "inspection" in the contract-theory sense. The theory predicts the minimum audit rate and the payment sizes needed to make genuine critique the model's best response. These are concrete thresholds that can be confirmed or refuted on planning and math benchmarks.
Core Claim
Consider training on reward for final correctness as judged by a model. Suppose the judge rejects correct answers with false-negative rate $f$. A correct answer then earns only $(1-f)$ of the reward. Revising it, which re-rolls the judge's verdict, can pay as much or more. On Mystery Blocksworld, GPT-4 used as a verifier had a 97.09% false-negative rate (On the self-verification limitations of large language models on reasoning and planning tasks). At that rate, final-correctness pay from such a judge gives the model essentially no reason to keep a correct plan. The opposite failure is also cheap. A policy that never revises collects final-correctness pay without spending tokens on critique.
Our claim is that a small transition-contingent contract, combined with random sound inspection at rate $\beta$, can make "keep if correct, fix if wrong" the policy's best response. The required $\beta$ and payments can be computed from the judge's measured error rates and the token cost of revising. Matching full sound supervision should be possible at a small fraction of its verification cost.
To make the objective precise, we reuse the two-parameter model of A Probabilistic Inference Scaling Theory for LLM Self-Correction. In that model, $CL$ is the probability a correct answer survives a revision round and $CS$ is the probability a wrong answer gets fixed. Accuracy then converges to
$$Upp = \frac{CS}{1 - CL + CS},$$
regardless of starting accuracy. That paper treats $CL$ and $CS$ as fixed properties of a model. We treat them as the agent's chosen behavior, which the principal (the trainer) shapes through payments, and we take the principal's goal to be maximizing $Upp$ net of verification cost. Write $p_{o}$ for the payment on observed transition $o$ and $c(a)$ for the token cost of action $a$. The behaviors $a$ are keep, revise, and abstain/declare unsolvable. Under inspection rate $\beta$, the incentive to keep a truly correct answer rather than revise it is
$$\Delta(\beta) = \beta\,(p_{\text{keep}\mid\text{correct}} - \mathbb{E}[p\mid\text{revise, correct}]) + (1-\beta)\,(\text{same gap under judge noise}) + c(\text{revise}) - c(\text{keep}).$$
Here the expectation is over the outcome of revising a correct answer. Requiring $\Delta(\beta) \ge \delta$ yields a closed-form minimum audit rate. The margin $\delta$ protects against the policy being only an approximate best responder, a robustness notion taken from Contract Design Under Approximate Best Responses. A parallel constraint makes fixing a wrong answer worth its token cost.
Why contract theory, and why only a little of it
The complexity results in this field constrain the design. The Complexity of Contracts shows that optimal contracts are tractable when the agent has a constant number of actions. Even constant-factor approximation becomes NP-hard once actions are arbitrary. Bayesian agency: Linear versus tractable contracts shows that with unknown agent types, no tractable contract beats simple linear pay by much.
We take these as design rules rather than obstacles. We do not try to contract over the model's full token-level policy, where the hardness results bite. We contract over a handful of coarse behaviors (keep, revise, abstain, optionally "think longer"), which the trainer can label after the fact from the transcript. Over that small action set, the optimal transition contract is a small linear program. The Bayesian hardness results also justify treating linear pay as a serious competitor rather than a strawman.
Two papers supply the method. The inspection mechanism comes from Contract Design With Safety Inspections, which pairs a linear payment share with random audits under a budget and finds optimal single-agent contracts in $O(n \log n)$ time. That model assumes the audit is sound and the only verdict available. We add a second, cheaper but noisy verdict source used on non-audited episodes. This is the change that matters for language models. The adversarial margin $\delta$ comes from Bacchiocchi et al., whose model assumes errors are worst-case deviations near the optimum. We instead calibrate the noise from measured judge error and measured $(CL, CS)$.
Research Program
The first step is measurement. For each task and base model, we estimate the model judge's false-positive and false-negative rates against the sound checker. We also estimate base $(CL, CS)$ under prompting and the token cost of each behavior. The second step is design. We solve the small linear program for transition payments and minimum $\beta$ across a range of audit costs. We also derive the comparative statics, meaning how the optimal payments and audit rate shift as conditions change. For example, by analogy to the "Safety Cost Paradox" of Fallah and Jordan, raising the cost of revising should push the optimal contract toward larger payments and fewer audits. The third step is training. We run reinforcement learning on open models with the designed reward, using a group-relative policy-gradient method of the GRPO type, applied identically across all arms. The models are Llama3-8B-Instruct and Qwen2.5-7B, which Yang et al. already used for $(CL, CS)$ estimation. The fourth step is checking. We test whether the trained policy's realized $(CL, CS)$ moves where the theory predicts, and whether the realized $Upp$ matches the formula.
The tasks are chosen so a sound checker exists for evaluation even where we deliberately ration it during training. They are Blocksworld and Mystery Blocksworld from PlanBench with a plan validator, the unsolvable-instance set described in LLMs Still Can't Plan; Can LRMs?, Graph Coloring, Game of 24, and GSM8K. The unsolvable set makes the abstain action meaningful. On that set o1-preview identified unsolvability only 27% of the time and produced a full invalid plan 54% of the time, which is exactly the kind of failure a payment for honest abstention targets.
Baselines
The naive reference point is prompted intrinsic self-correction with no training. Large Language Models Cannot Self-Correct Reasoning Yet documents this degrading accuracy. On GSM8K, Llama-2-70b fell from 62.0% to 36.5% and GPT-4 from 95.5% to 89.0%. We rerun it on our base models with the same prompts used at evaluation for every other arm.
The strongest inference-only competitor is sampling with a sound checker. In Stechly et al., "Sound Verifier + Sampling ($k=25$)" with GPT-4 reached 72% on Blocksworld, 14% on Mystery Blocksworld, 44% on Graph Coloring, and 42% on Game of 24. On GSM8K, Huang et al. report Self-Consistency, meaning independent samples with a majority vote, at 88.2% with 9 calls, against 83.0% for multi-agent debate. We run both on our base models and on our trained models with matched model calls and checker calls. Beating them means higher accuracy at equal total compute, or equal accuracy at lower compute. Those GPT-4 figures are reference points from a different model and are not directly comparable to our 7–8B runs.
The strongest training competitor is final-correctness reward with the sound checker on every episode ($\beta = 1$). This is the upper bound our method aims to approach. Success means reaching at least 95% of its realized $Upp$ at $\beta \le 0.2$, with the same RL algorithm, steps, and hyperparameter search budget. We found no reported result on Lacuna for RL-trained self-correction on these benchmarks, so we establish this number ourselves.
The weaker training competitor is final-correctness reward from the model judge alone ($\beta = 0$). Next is a linear contract: pay proportional to judged final correctness, with a constant token penalty and the same $\beta$ audits.
The key ablation keeps everything about our method, including judge, $\beta$, and total reward scale, but pays on the final outcome rather than on the transition. This isolates whether transition-contingent pay is the active ingredient.
Why not just…?
Why not just audit every training episode with the sound checker? On PlanBench the validator is cheap, so for these tasks this is a real threat, and we include it as the upper-bound arm. The method matters in the regime the benchmarks only simulate, where sound checking is expensive or available for a small slice of data. Examples are long-horizon plans, or deployments where the only scalable judge is another model. We emulate that regime by charging a per-audit cost and sweeping it. If full auditing wins at every cost we consider, the contribution reduces to the measurement study, and we will report it that way.
Why not just denoise the model judge's reward using its measured error rates? An unbiased estimate of final correctness is achievable this way, and it is a genuine competitor that we run under the same budget. It does not change which behavior is cheapest, though. Keeping an answer is cheaper in tokens than critiquing it, so a correctly estimated final-correctness reward still favors a never-revise policy whenever the expected gain from fixing is below the token cost. That policy has $CS \approx 0$ and does not improve over rounds. The transition-versus-final ablation tests this directly. If denoised final reward produces the same $(CS, CL)$ as our contract, the contract framing adds nothing.
Why not just skip training and use self-consistency or sample-and-check at inference? Huang et al. and Stechly et al. show these beat intrinsic critique at matched calls, so they are the bar. They cannot help on unsolvable instances, where no sample passes the checker. They also spend a fixed $k$ calls per problem. On Large-Scale Blocksworld, even o1-preview reached only 23.6%, so small $k$ will often find no valid plan. We compare at matched calls. If sample-and-check dominates our trained models at every budget, we will conclude that incentive design for revision is not worth it.
Why not just use a linear contract, which theory says is near-optimal? Castiglioni et al. indeed show tractable contracts rarely beat linear pay. That guarantee concerns an agent with unknown type facing outcome-based pay. Here the problem is a different one: a noisy verdict makes keeping and breaking correct answers pay alike. We therefore expect the gap between linear and transition pay to grow with judge noise and to vanish when the judge is accurate. Running both on standard and Mystery Blocksworld, which have very different judge error, tests this.
Decisive Checks
Three results would confirm the proposal and their absence would refute it. First, the theory predicts a minimum audit rate $\beta^*$ from measured judge error. Policies trained just below $\beta^*$ should show a sharp drop in $CL$, keeping fewer correct answers, and policies trained just above should not. We test this by sweeping $\beta$ on Mystery Blocksworld, where judge false negatives are extreme, and on GSM8K, where they should be mild, so the predicted thresholds differ by a large margin. Second, $Upp$ computed from the trained policy's measured $(CL, CS)$ should match its realized multi-round accuracy, as Yang et al.'s convergence formula predicts. A mismatch would mean the coarse action model misses what the policy actually does. Third, on PlanBench's unsolvable instances, a positive payment for verified abstention should raise correct unsolvability detection well above the base model's rate without increasing false abstention on solvable instances. Throughout, we report accuracy, realized $CL$ and $CS$, and checker calls per solved problem.
References
- On the self-verification limitations of large language models on reasoning and planning tasks
- A Probabilistic Inference Scaling Theory for LLM Self-Correction
- Contract Design Under Approximate Best Responses
- The Complexity of Contracts
- Bayesian Agency: Linear versus Tractable Contracts
- Contract Design With Safety Inspections
- PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Change
- LLMs Still Can't Plan; Can LRMs? A Preliminary Evaluation of OpenAI's o1 on PlanBench
- Large Language Models Cannot Self-Correct Reasoning Yet