Choosing Where to Stop Imitating: Label-Free Weight Interpolation for Weak-to-Strong Fine-Tuning

When a large language model is fine-tuned on labels produced by a weaker model, it learns to copy the weaker model's mistakes. This is the "imitation trap": the strong model's accuracy is dragged down toward its teacher's. A separate literature shows that fine-tuning also erodes robustness to distribution shift, and that averaging the weights of the pretrained and fine-tuned models recovers much of that robustness. A third literature treats a frozen model's few-shot behavior as implicit Bayesian inference: the examples in the prompt act as evidence that picks out one task from many the model already knows. We propose to join these three through one testable claim. Weak-supervised fine-tuning does not destroy the strong model's knowledge of the right answer. It shifts the model's belief about which task it is being asked to do, toward "reproduce the weak teacher." If that is true, the fix is to find the point on the straight line between pretrained and fine-tuned weights where the model still does the intended task. That point can be located without gold labels or out-of-distribution data, by checking the model against the pretrained model's own few-shot beliefs on the examples where those beliefs disagree with the teacher.

In two sentences for a colleague: fine-tuning on a weak teacher's labels pushes a strong model to impersonate the teacher, but the correct answers are still latent in its weights. We show this directly, then pick the weight-interpolation coefficient by measuring agreement with the base model's few-shot predictions only on the inputs where the base model confidently disagrees with the teacher. This addresses both the imitation ceiling and the open problem of setting the interpolation coefficient without out-of-distribution validation data.

Core Claim

The claim has two parts. The first is diagnostic. The imitation trap described by Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision, where strong students overfit systematic weak-teacher errors, is the same phenomenon as the prior skew documented in Understanding Catastrophic Forgetting in Language Models via Implicit Inference. That paper shows fine-tuning does not erase capabilities. It re-weights the model's belief over tasks, and prompts moved away from the fine-tuning format recover what was suppressed: instruction-tuned Alpaca lost 56.75% of English in-context accuracy but only 1.5% under Leetspeak. That paper studied clean instruction data. Nobody has asked whether weak labels produce a skew specifically toward the teacher's error pattern, one that is recoverable rather than a real loss of the knowledge needed to answer correctly.

The second part is a method. Weight-space interpolation, $\theta_\alpha = (1-\alpha)\theta_0 + \alpha\theta_1$ as in Robust fine-tuning of zero-shot models, traces a path from the pretrained weights $\theta_0$ to the weak-supervised weights $\theta_1$. That paper reports that choosing $\alpha$ still depends on a default of 0.5 or on out-of-distribution validation data. We choose $\alpha$ using a signal specific to weak supervision.

Let $p_{\text{ICL}}(y \mid x)$ be the frozen base model's predictive distribution when prompted with $k$ weak-labeled demonstrations. This is the "ICL refinement" signal of A transfer learning framework for weak to strong generalization. Let $\mathcal{D}$ be the unlabeled training inputs where $p_{\text{ICL}}$ is confident (maximum probability above a threshold $\tau$) and its top answer differs from the weak label. We select

$$\alpha^\star = \arg\max_{\alpha} \frac{1}{|\mathcal{D}|}\sum_{x \in \mathcal{D}} \mathbb{1}\left[\arg\max_y p_{\theta_\alpha}(y\mid x) = \arg\max_y p_{\text{ICL}}(y \mid x)\right],$$

subject to $\theta_\alpha$ keeping agreement with weak labels on the complement of $\mathcal{D}$ within a small tolerance of $\theta_1$. The logic is that imitation shows up exactly on $\mathcal{D}$. Off $\mathcal{D}$, teacher and base model agree, so fitting the weak labels there is task learning rather than error copying.

Research Program

The program has three stages.

The first stage tests whether the capability survives. We train strong students on weak labels and track four things across training:

That last paper showed supervised fine-tuning collapses middle-layer representations into label clusters while in-context learning preserves input structure. It used only 2-D synthetic tasks and clean labels. We measure whether collapse under weak labels organizes hidden states around the teacher's labels rather than the true ones.

The second stage traces the interpolation path and asks whether disagreement-set agreement predicts true and out-of-distribution accuracy along it. The third stage turns the rule into a method, including per-layer coefficients, and stress-tests it under confident, systematic teacher errors.

Baselines

The naive reference point is standard fine-tuning on weak labels, early-stopped on weak-label validation loss. Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision reports it recovers about 50% of the performance gap across NLP benchmarks including SciQ. The gap is measured as performance gap recovered (PGR): the fraction of the distance between weak-teacher and gold-trained-student accuracy that the method closes. We also report the frozen base model's few-shot accuracy with weak demonstrations. This is the anchor signal itself, and any method must beat it to justify fine-tuning at all.

The strongest single-teacher competitor is the auxiliary confidence loss from the same paper, reported at nearly 80% PGR across NLP tasks. Alongside it we run the reverse-KL and reverse cross-entropy objectives of Revisiting Weak-to-Strong Generalization in Theory and Practice: Reverse KL vs. Forward KL. That paper reports these beat forward objectives in 12 settings and tolerate up to 50% label swaps. We also run ICL-refinement relabeling followed by fine-tuning, from A transfer learning framework for weak to strong generalization.

On SciQ the strongest reported number is the evidential multi-teacher method of Bayesian WeakS-to-Strong from Text Classification to Generation, at PGR 0.828 with five weak models versus 0.614 for the best single weak model. Because our method uses one teacher, the fair comparison is against 0.614 for the single-teacher setting. We report 0.828 as the ceiling a single-teacher method would need to approach. Under shift, the reference is RAVEN from Weak-to-Strong Generalization under Distribution Shifts, which reports a 57% average PGR improvement on out-of-distribution text classification over ensemble baselines. For weight interpolation in language-model weak-to-strong settings, no reported result was found on Lacuna, so we establish it ourselves.

Three ablations isolate the key ingredient, which is the disagreement-set selector:

  • interpolation at the default $\alpha = 0.5$;
  • interpolation with $\alpha$ chosen on gold out-of-distribution labels, an oracle upper bound;
  • interpolation with $\alpha$ chosen by agreement with $p_{\text{ICL}}$ over all inputs rather than only $\mathcal{D}$.

All methods share the same weak labels, strong model, fine-tuning budget, and the same number of hyperparameter trials. Selection uses only weak-labeled and unlabeled in-distribution data, except the oracle. Beating the field means two things. First, higher out-of-distribution PGR than the confidence loss and reverse KL, with in-distribution PGR within one point of the best. Second, closing most of the gap between default and oracle $\alpha$.

Why not just…?

Why not just use weight interpolation at $\alpha = 0.5$? This is a real threat. If the fixed default already sits near the oracle on most tasks, our selector adds nothing. The default-versus-oracle-versus-selected comparison settles it directly. We expect the gap to be largest when teacher error rates vary across tasks, because the right amount of pull back toward $\theta_0$ should scale with how much imitation occurred. The robust fine-tuning paper itself flags coefficient choice as unresolved.

Why not just relabel with the base model's few-shot predictions and fine-tune on those? This is ICL refinement from A transfer learning framework for weak to strong generalization, and it is a baseline. Two things argue it is insufficient alone. Fine-tuning on any labels still collapses representations, per the in-context versus fine-tuning comparison. And that paper reports its gains become modest on MATH, where few-shot predictions are poor. Our method uses the few-shot signal only to choose a point on the interpolation path, so the signal need not be correct everywhere. If relabel-then-fine-tune matches us out of distribution, we will report the selector as a coefficient-choice tool rather than an imitation fix.

Why not just early-stop, or use LP-FT? LP-FT trains the classification head first with the backbone frozen, then fine-tunes everything. Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution states that no point on the standard fine-tuning trajectory matches LP-FT out of distribution, and Andreassen et al. show early stopping sacrifices in-distribution accuracy. LP-FT itself fits its head to the weak labels, aligning it with teacher errors. We include both early stopping and LP-FT under the shared budget. We also test whether the weak-to-strong accuracy peak coincides with the effective-robustness peak. If it does, early stopping becomes a much stronger competitor than the literature suggests.

Proposed Experiments

Weak and strong pairs come from the Pythia suite, used in the reverse-KL study, for controlled scale sweeps. One larger open family of the kind used in the representation-collapse study supplies stronger few-shot inference. In-distribution tasks are SciQ and CosmosQA. The distribution-shift task is Amazon-WILDS text classification. A reward-modeling setting on HH-RLHF probes the regime where Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision reports naive PGR of only 10–20%. Metrics are in- and out-of-distribution PGR, effective robustness, recovery rate on the teacher-error subset, and a representation-collapse score from representational similarity analysis.

Three checks decide the question.

The first is recoverability. If gold few-shot or conjugate prompts on the weak-supervised checkpoint recover a large share of accuracy on teacher-error inputs, the prior-skew account holds. If they recover little, the capability was actually overwritten and the method's premise fails.

The second is predictive validity. Across the interpolation path, the rank correlation between disagreement-set agreement and gold out-of-distribution accuracy must be clearly positive on every task. A near-zero correlation falsifies the selector.

The third is a confident-bias stress test. Following the tension raised by A Closer Look at Model Adaptation using Feature Distortion and Simplicity Bias, we give the weak teacher a confident, systematic spurious rule, such as a lexical cue tied to one answer. We predict reverse KL degrades sharply here because it locks onto confident modes. Our selector should degrade only if the base model's few-shot inference also latches onto the cue. We measure this by checking whether $p_{\text{ICL}}$ follows the cue. That result tells us where implicit Bayesian inference can and cannot serve as a check on a weak teacher.