When a strong model is trained on labels from a weaker one, it tends to copy the teacher's mistakes and ends up near the teacher's accuracy rather than its own. This happens even though its pre-trained knowledge could have corrected many of those mistakes. The most effective simple fix so far lets the student override the teacher by a fixed amount everywhere. We argue that the amount of overriding should instead be measured. With a few hundred trusted labels, conformal calibration can estimate how far the teacher's predictions sit from the truth in different regions of the input space. The student is then allowed to move away from the teacher only within that measured distance: freely where the teacher is shown to be unreliable, barely at all where it is shown to be reliable.
The mechanism uses the language of distributionally robust optimization (training against a set of plausible data distributions rather than one). We turn it around. The usual robust objective is pessimistic: it upweights examples where the model's loss is high. Here, high loss against weak labels marks exactly the examples where the student resists the teacher, which are often the places where the student is right. So we predict that the usual pessimistic objective speeds up imitation. The useful object is an optimistic, calibrated set of labels around the teacher that the student may choose from.
Core Claim
Write $p_w(\cdot\mid x)$ for the weak teacher's predicted label distribution on input $x$ and $p_s$ for the student's. Define a trust region $U(x)=\{q : \mathrm{TV}(q, p_w(\cdot\mid x)) \le \rho(x)\}$, where TV is total variation distance and $\rho(x)$ is a radius. The student minimizes
$$\min_\theta \; \mathbb{E}_x \Big[\min_{q \in U(x)} -\textstyle\sum_y q(y)\log p_s(y\mid x;\theta)\Big].$$
The inner problem is linear in $q$, so it has a closed form: move up to $\rho(x)$ of probability mass from the classes the student finds least likely onto the student's top class. The key observation is that the auxiliary confidence loss of Weak-to-Strong Generalization is a special case of this objective. That loss mixes the weak label with the student's hardened prediction at weight $\alpha$, which is the same as a single global radius $\rho=\alpha$ applied identically to every input.
Our proposal replaces that global constant with an input-dependent radius. The radius is calibrated on a small gold-labeled set $\{(x_i, y_i^*)\}$ using the score $s_i = 1 - p_w(y_i^*\mid x_i)$, which is the TV distance from the teacher's prediction to the true one-hot label. Scores are computed separately within groups $g(x)$ formed by crossing two binned quantities: teacher confidence, and disagreement between teacher and the pre-fine-tuning student. For a coverage level $1-\alpha$, group-wise conformal quantiles give radii $\rho_g$ such that, with probability at least $1-\alpha$ within each group, the true label lies inside $U(x)$.
This split by disagreement is what should handle the case Reverse KL cannot: confident teacher errors. If the calibration data show that a confident teacher is often wrong in cases where the strong model disagrees with it, then that group receives a large radius, however confident the teacher is.
The guarantee is about the trust region, not the student's final accuracy, and we do not claim more. What we claim empirically is that calibrated, input-dependent radii recover more of the gap between teacher and student than a tuned global radius. We also expect the advantage to be largest when teacher errors are systematic and confident.
Research Program
The work has three parts.
Build. We will implement the calibrated trust-region loss for classification, and then for pairwise preference labels, treating the preferred-response indicator as a binary label.
Analyze. We will relate the method to the theory in Understanding the Capabilities and Limitations of Weak-to-Strong Generalization. That paper bounds the student's gain over the teacher by $O(\sqrt{d_P})$, where $d_P$ is student–teacher disagreement. It also shows that students inherit the teacher's calibration error. Its role here is motivation: it identifies disagreement as the resource that drives gains, but it proposes no way to spend that resource selectively. Our radius is a disagreement budget allocated by measured teacher error. We will test whether the budget actually spent concentrates where the teacher is wrong on gold data, and whether this reduces the inherited expected calibration error (ECE, the average gap between stated confidence and actual accuracy) that Yao et al. report.
Calibrate. The machinery comes from uncertainty-set calibration. A Unified Perspective on Conformal Prediction and Wasserstein DRO shows that conformal quantiles and robust radii are two ways of correcting the same quantile estimate. It also shows that metric radii on discrete label spaces can become vacuous, covering every label. We avoid that failure by placing the ball in TV distance over the teacher's probabilities, not in a metric over labels. The paper is inference-only; we use calibration to shape training.
To choose $\alpha$ without selection bias, we adopt the two-split procedure of Calibrating Decision Robustness via Inverse Conformal Risk Control. That method certifies a trade-off curve between risk and conservatism across a family of set sizes, and so far it has been tested only on small classical optimization problems. In our setting, the risk is miscoverage of the true label and the cost is the average radius, which measures how much freedom the student gets. Our contribution is to carry it into neural fine-tuning.
Conformal Margin Risk Minimization is the closest method in spirit. It applies per-batch conformal thresholds on logit margins to downweight noisy labels. It calibrates without trusted labels, has no notion of a stronger student's prior, and was tested only on vision label noise. We differ by calibrating teacher error against gold labels, conditioned on the student's own disagreement.
Baselines
The naive reference is standard cross-entropy fine-tuning on weak labels, early-stopped using the gold set. The strong model is also trained on ground truth to supply the ceiling for Performance Gap Recovered (PGR, the fraction of the teacher-to-ceiling accuracy gap the student closes). Weak-to-Strong Generalization reports that naive fine-tuning recovers about 50% PGR on its NLP suite and only about 10–20% on reward modeling.
The strongest existing competitor is auxiliary confidence loss, which reaches nearly 80% PGR on the same NLP suite in that paper. It is the strongest because it is exactly our objective with a constant radius. We will give it the same gold set, used to tune $\alpha$ over a grid, and match the number of training runs we use to select our coverage level. Beating it means higher mean PGR across tasks with non-overlapping seed intervals, and especially clear gains on the confident-bias variants described in the experiments.
Reverse KL / Reverse Cross-Entropy is run as specified in Revisiting Weak-to-Strong Generalization: Reverse KL vs. Forward KL, including its combination with confidence regularization. That paper reports gains in 10 of 12 settings, but the file lists no tabulated numbers for it.
Pessimistic TV-DRO is the reweighting of Distributionally Robust RLHF, applied around the weak labels, with $\rho$ swept over their reported useful range of 0.1 to 0.2. That paper reports a RewardBench reasoning score rising from 66.8 to 71.4 at $\rho=0.2$. That result concerns prompt shift with human labels, not a weak teacher, so it is not a direct comparison.
Gold-data baselines use the calibration labels for training instead: once on gold labels alone, and once on weak plus upweighted gold labels.
Three ablations isolate the key ingredients. The first uses one calibrated global radius. The second groups by teacher confidence only, without student disagreement. The third uses the conditional radii but computes disagreement from the evolving student rather than the frozen initial one, which tests for feedback loops.
Why not just…?
Why not just tune the auxiliary confidence weight on the gold set? This is the main threat. If a tuned global $\alpha$ matches us, then the input-dependent radius adds nothing and the contribution reduces to a reinterpretation of Burns et al. The global-radius ablation settles the question directly. We expect a gap only where teacher reliability varies sharply across groups, and our confident-bias splits are built so that it does. If no gap appears there, we will report it as a negative result.
Why not just train on the gold labels? Estimating a handful of group quantiles needs far fewer labels than fitting a classifier, so calibration should win when gold labels are scarce. It may lose once gold data are plentiful. We sweep the gold set size from 100 to 3000 and report where the curves cross, rather than asserting dominance.
Why not just use Reverse KL? Its mode-seeking behavior follows the teacher's confident modes. The authors themselves list confident bias as an open problem. Our injected confident-error splits test whether Reverse KL locks onto those errors while calibrated radii escape them.
Why not just use standard distributionally robust training? Pessimistic reweighting upweights high-loss examples, and against weak labels these are the student's correct disagreements. We predict it lowers PGR below naive fine-tuning as $\rho$ grows, and the sweep checks this directly.
Proposed Experiments
Classification. We use the NLP suite from Burns et al. (BoolQ, SciQ, HellaSwag and related tasks) and CAI-Harmless, with teacher/student pairs drawn from the GPT-2, Pythia and Qwen2.5 families as in Yao et al. We report PGR, gold accuracy, ECE, and the fraction of trust-region budget spent on examples where the teacher is wrong. The file has no reported numbers for CAI-Harmless.
Confident-bias stress test. We create teacher errors in two ways. One trains the weak teacher on data where a spurious keyword predicts the label. The other flips a subgroup's labels at high confidence, extending the 50% label-swap setting of the Reverse KL paper. This is the decisive test of the conditional radius.
Reward modeling. Weak reward models label HH-RLHF pairs, strong reward models are trained on those labels, and evaluation is on RewardBench. The file has no reported weak-to-strong result on RewardBench. Chess puzzles serve as a hard case where Burns et al. saw little transfer; no reported PGR number was found on Lacuna for chess.
Coverage and validity checks. We report empirical coverage of the true label on held-out gold data, per group, against the $1-\alpha$ target. We also report how PGR varies with $\alpha$ along the certified trade-off curve.
The proposal succeeds if three things hold together. It must beat tuned auxiliary confidence on the confident-bias splits. It must hold coverage near target. And pessimistic robust training must be shown to hurt, as predicted.
References
- Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision
- Understanding the Capabilities and Limitations of Weak-to-Strong Generalization
- A Unified Perspective on Conformal Prediction and Wasserstein Distributionally Robust Optimization for Uncertainty Quantification
- Calibrating Decision Robustness via Inverse Conformal Risk Control
- Conformal Margin Risk Minimization: An Envelope Framework for Robust Learning under Label Noise
- Revisiting Weak-to-Strong Generalization in Theory and Practice: Reverse KL vs. Forward KL
- Distributionally Robust Reinforcement Learning from Human Feedback