Tail-Calibrated Gates for LLM Self-Correction

When a language model is asked to critique and rewrite its own answer, the rewrite usually changes nothing, sometimes fixes a real mistake, and occasionally breaks an answer that was already right. These rare breakages decide whether self-correction helps or hurts overall. Systems that let the model revise every time, or that trigger a rewrite whenever the model's own uncertainty looks high, are exposed to them. On code, uncertainty-triggered regeneration lowered accuracy in five of six configurations (When Uncertainty Isn't Enough). We propose to treat the decision "keep this revision, discard it, or pay for an expensive check" as a problem about the tails of a distribution. A small model reads cheap signals from the generate–critique–revise loop and predicts the full distribution of how much the revision changes answer quality, not a single probability of correctness. Its tails are allowed to be heavier or lighter than a bell curve, and it is calibrated only in the harmful tail where the decision is actually made.

The distribution family is the generalized Gaussian, a bell curve with an extra knob that controls tail weight. We borrow the idea of calibrating only the decision-relevant tail from Bayesian optimization, where Goal-Oriented Lower-Tail Calibration of Gaussian Processes showed that matching only the lower tail of a surrogate's residuals, using a generalized normal fit, beats global calibration for choosing where to sample next. The bet is concrete and can fail. Revision-induced quality changes are sharply peaked at zero with heavy, asymmetric tails. A parametric tail model fit on a few hundred calibration problems should therefore estimate the probability of a harmful revision better than distribution-free alternatives, which have almost no data at extreme quantiles. Better tail estimates should then translate into a better accuracy-per-compute frontier than both blind refinement and always-verify.

Core Claim

Let $\Delta = q(\text{revised}) - q(\text{original})$ be the change in a continuous quality score $q$ produced by one round of self-correction. For code, $q$ is the fraction of hidden unit tests passed. For math, $q$ is the step-level correctness score of a strong external process verifier. Given cheap features $x$ of the round, we model

$$p(\Delta \mid x) \propto \exp\left(-\left|\frac{\Delta - \mu(x)}{\alpha(x)}\right|^{\beta(x)}\right),$$

where $\mu$ is the expected change, $\alpha>0$ its scale, and $\beta>0$ its tail shape. The value $\beta=2$ gives a Gaussian, and $\beta<1$ gives a sharp peak with heavy tails. The cheap features include the token entropy of the revision, the probability of a "this is correct" token in a ReVISE-style model, whether the critique asserts a specific error, the size of the edit, and the round index.

The gate keeps a revision only when the predicted probability of harm is small:

$$\Pr(\Delta < -\tau \mid x) \le \varepsilon.$$

Here $\tau$ is the smallest quality drop we count as a regression and $\varepsilon$ is a user-set risk budget. When harm and benefit are both plausible, the gate escalates to an expensive check such as unit-test execution or the external verifier. Otherwise it reverts to the original answer. The same rule decides whether a multi-round loop should continue.

The parameters $\mu$, $\alpha$, and $\beta$ are fit as in tcGP. Shape and scale are tuned only on calibration points below a data-driven quantile threshold. The fit targets two things: getting the frequency of harmful revisions right (occurrence calibration) and getting the conditional distribution within that tail right. Overall goodness of fit is not the objective.

This differs from each piece of prior work in a specific way. When Uncertainty Isn't Enough is the motivating negative result. It found that raw entropy, verbalized confidence, and probes make poor regeneration triggers, and it recommended using cheap signals as an asymmetric gatekeeper for expensive verification. It did not build or calibrate such a gate, and it never modeled the quality change itself. ReVISE uses the softmax probability of a single verification token as confidence. We use that probability as one input feature and replace the point estimate with a calibrated distribution over outcomes. CritiCal improves verbalized confidence by training on teacher critiques and measures success with binned calibration error. We do not touch the generator. We calibrate a separate decision model, and we score it on tail risk rather than average calibration. The method itself comes from ProbVLM, which showed that a lightweight adapter predicting generalized Gaussian location, scale, and shape on top of frozen foundation-model outputs yields well-ranked uncertainty (Spearman −0.99 against retrieval quality on COCO). It also comes from the image-compression work of Zhang et al., whose scale lower bounds and gradient rectification we reuse to keep $\beta$ from collapsing toward 0 or diverging during training, a known instability of joint scale–shape fitting.

Research Program

Data collection. We first collect self-correction trajectories at scale. The code models are Llama-3.2-3B, Qwen2.5-Coder-3B, and DeepSeek-Coder-1.3B on HumanEval, MBPP, and BigCodeBench. The math model is Qwen2.5-1.5B on MATH500 and AMC, run in the breadth-by-depth loop of Refining Over Resampling, in which each of $N$ rollouts undergoes $D$ rounds of self-critique and revision. Every round is logged with its cheap features and its true $\Delta$. For math we use GenPRM-7B as the scorer, since GenPRM reports a ProcessBench F1 of 80.5 for it. We also log final-answer correctness as ground truth.

Testing the premise. Before building anything, we check whether the bet holds. We examine the empirical distribution of $\Delta$ and the fitted $\beta$ values. If revision effects turn out roughly Gaussian ($\beta\approx 2$ throughout), the shape knob is unnecessary and we report that. The same is true if they are essentially three-valued (no change, full fix, full break): a three-class classifier is then the right tool.

Building and deploying the gate. We then train the gate head, a small network over the features, in four variants: fixed Gaussian, fixed Laplace ($\beta=1$), learned per-instance $\beta$, and learned $\beta$ fitted globally rather than in the tail. Finally we deploy each gate inside the loops and measure accuracy, the rate at which correct answers are revised into incorrect ones, and total compute. Compute counts generation tokens, verifier calls, and test executions under a single cost model.

Baselines

The naive reference points are single-pass generation and always-accept refinement. Always-accept means Self-Refine-style loops (Self-Refine) for code and the breadth-depth loop for math. On MATH500 that loop reported 58.0% for Qwen2.5-1.5B against 29.6% for the strongest verifier-based Best-of-N. Any gate must at least match always-accept accuracy while cutting compute or regressions.

The first baseline from the negative-result literature is uncertainty-triggered regeneration as implemented by When Uncertainty Isn't Enough. That paper reports it dropping Llama-3.2-3B HumanEval Pass@1 by nearly 10 points. It runs on identical trajectories and features.

The strongest existing method, and our compute ceiling, is verification-based regeneration that always executes tests. The same paper reports DeepSeek-Coder-1.3B on HumanEval rising from 0.61 to 0.87 under this scheme. For math, the equivalent is consulting GenPRM-7B on every round. Beating it means reaching within one point of its accuracy while invoking the expensive check on substantially fewer rounds. We report the full accuracy–cost curve, not one operating point.

For MBPP the strongest trained-model reference is ReVISE at 33.1% Pass@1. Its confidence-aware sampling serves as a gating baseline on the math tasks where we can reproduce it.

Three distribution-free gates use exactly the same features, calibration split, and hyperparameter search budget:

  • a logistic or isotonic classifier predicting "harmful revision";
  • quantile regression on $\Delta$;
  • split conformal prediction intervals on $\Delta$.

These are the real competitors on this question.

The key ablations hold the network fixed and vary only the tail treatment. We fix $\beta=2$, fix $\beta=1$, and fit learned $\beta$ globally rather than in the tail. If learned tail-calibrated $\beta$ does not beat all three, the flexible-shape ingredient is not doing the work.

Why not just…?

Why not just train a classifier for "this revision breaks a correct answer" and threshold it? This is the most serious threat. If harm is effectively binary, a calibrated classifier is all one needs. We address it in two ways. First, for code and verifier-scored math, $\Delta$ is continuous, and the gate's three-way decision of accept, revert, or escalate needs the joint shape of harm and benefit, not one probability. Second, the head-to-head comparison is run at matched features and calibration data. If the classifier matches our gate, we report that tail shape does not matter here.

Why not just use quantile regression or conformal intervals, which need no distributional assumption? These are the right default when data is plentiful. The gate operates at small risk budgets ($\varepsilon$ of a few percent), where a calibration set of a few hundred problems contains only a handful of harmful revisions. A three-parameter tail model can extrapolate there, while empirical quantiles cannot. tcGP's speed and stability came from exactly this kind of parametric tail fitting, although its guarantees assume noiseless evaluations and ours are noisy. We test the claim directly by sweeping calibration-set size from 100 to 2,000 problems. If quantile regression and conformal intervals catch up at moderate sizes, the honest conclusion is that the generalized Gaussian helps only in the low-data regime.

Why not just always run the tests or the verifier? Where tests exist, this is accurate. The 0.61→0.87 result shows it, and it is our ceiling. But it costs an execution or a large verifier call on every round, and no such oracle exists for open-ended tasks. The gate is designed to defer to it selectively. Its value is measured as the fraction of verifier calls saved at matched accuracy.

Why not just sample more and vote? Refining Over Resampling documents diversity saturation, where extra samples repeat the same error modes. It also shows refinement beating Best-of-N by large margins on MATH500 and AMC. Resampling is still included as a compute-matched point on the frontier.

Decisive Checks

The idea stands or falls on three measurements made on held-out problems. The first is the fitted distribution of $\beta$: a value near 2 everywhere ends the shape argument. The second is calibration of the harmful-revision frequency at small $\varepsilon$ against the distribution-free gates, across the calibration-size sweep. The third is the end-to-end accuracy-versus-compute frontier on HumanEval, BigCodeBench, MBPP, and MATH500. Success means the tail-calibrated gate dominates always-accept refinement, uncertainty-triggered regeneration, and the fixed-shape ablations. It should also cut correct-to-incorrect flips at least in half relative to always-accept, while using clearly fewer expensive checks than always-verify for equal accuracy.