Peer Distillation Across Prompt Variants to Reduce Instruction Brittleness

Large language models often change their answer when a question is reworded in ways that should not matter: the multiple-choice options are shuffled, a trailing space is added, "Thank you" is appended. On some benchmarks this prompt sensitivity accounts for up to half of all performance variance. Instruction-tuned chat models are often more sensitive than the base models they were trained from. The obvious training fix is to show one model many rewordings of each question and penalize it when its answers disagree. That fix has a hidden failure mode. A model can satisfy "agree with yourself" by becoming confidently and consistently wrong.

We propose to train two different language models together. Each model sees a different rewording of the same question, and each is pulled toward the other's answer distribution. Because the peer makes different mistakes, agreeing with it is a harder target than agreeing with oneself. The pairing we focus on is an instruction-tuned model with its own base model, which, unlike its tuned counterpart, has not had its sensitivity to wording increased by instruction tuning. This setup is called bidirectional knowledge distillation, or mutual learning: two networks train at the same time and each serves as the other's teacher. The question is whether this buys prompt robustness that single-model consistency training cannot, and whether the gain is explained by how differently the two peers err.

Core Claim

The claim is narrow and falsifiable. Cross-template co-distillation between two models with decorrelated errors reduces prompt sensitivity on held-out kinds of rewording more than single-model cross-template consistency training at equal compute. The gap should come mainly from fewer "consistent-but-wrong" answers. It follows that the benefit over self-consistency should grow with how decorrelated the peers' errors are. A peer that is a reseeded copy of the same model should give roughly nothing extra. A base/instruct pair should give more, and a pair from different model families should give the most, unless shared hallucinations dominate. If a reseeded copy works as well as a genuinely different peer, the two-model machinery is unnecessary and the honest conclusion is "use self-consistency."

Four lines of evidence motivate this. POSIX measures how much output log-likelihoods shift across 60 intent-preserving rewordings per prompt. It finds the instruction-tuning paradox (chat models more brittle than base models) and suggests its stability measure could become a training loss, but never builds one. ProSA ties brittleness to flat output distributions: where the model is unsure, small surface changes tip the top answer. On the Relation between Sensitivity and Accuracy in In-Context Learning shows that answers which wobble across rewordings tend to be wrong (Pearson $r$ down to −0.40). Together these say brittleness marks uncertainty, so penalizing disagreement without an independent signal of correctness risks rewarding confident error.

Two prior methods supply the mechanism. Co-training and Co-distillation (CTCD) trains two BERT models that each supply soft targets to the other. With it, a 4-layer student reaches 79.12 average GLUE versus 78.06 for the standalone 6-layer teacher. Bi-KD swaps teacher and student roles per example between diverse pretrained vision models, and reports +1.385% ImageNet top-1 for ViT-Base paired with ViT-Tiny. Neither touches prompt wording or generative language models. CTCD uses one fixed input format, and Bi-KD picks the teacher using ground-truth loss. Our change is to make each model see a different rewording of the shared input than its peer does. Prompt-invariance thus becomes the thing being transferred, rather than incidental regularization.

Research Program

For a training question $x$ with gold answer $y$, sample two templates $t_a, t_b$ from a pool of rewordings. Let $p_A(\cdot \mid t_a(x))$ be model A's distribution over answer options (or over short-answer tokens) under template $t_a$, and likewise $p_B(\cdot \mid t_b(x))$ for model B. Model A minimizes

$$ \mathcal{L}_A = \mathrm{CE}\big(y,\, p_A(\cdot \mid t_a(x))\big) + \lambda\, w_B(x)\, \mathrm{KL}\big(\bar p_B(\cdot \mid t_b(x)) \,\|\, p_A(\cdot \mid t_a(x))\big), $$

and B minimizes the symmetric loss. Here $\bar p_B$ is B's prediction with gradients stopped, $\lambda$ sets the strength of the peer term, and $w_B(x)\in[0,1]$ is a per-example weight on how much A should trust B.

The weight is where we depart from Bi-KD. Rather than letting ground-truth loss decide who teaches, $w_B(x)$ is B's own agreement across a small number of extra templates, computed without labels from a cheap extra forward pass. A peer teaches only where it is itself stable. This follows ProSA's finding that stable, sharply peaked predictions are the trustworthy ones. Because $w_B$ needs no labels, the same objective can later run on unlabeled instructions.

We train with LoRA adapters on 7B-scale open models so that two models fit in memory. Lee et al. report that co-distillation needed 20 epochs where standard distillation needed 10, so every comparison is matched on total training tokens and optimizer steps rather than epochs.

The analysis program has three parts. First, a peer-diversity sweep. We pair the instruct model with a reseeded copy of itself, with its base model, and with a model from another family. For each pair we measure error correlation on a held-out split, then test whether this correlation predicts the gain over self-consistency training.

Second, decomposing failures. We count errors that are consistent-but-wrong (same wrong answer under every template) versus unstable. The goal is to check that the mechanism is what we claim: self-consistency should shrink instability partly by increasing consistent-wrong answers, while peer distillation should not.

Third, separating output format from internal representation, the question raised by Flaw or Artifact?. That paper shows much measured sensitivity disappears under an LLM judge: Gemma-2.0 on ARC-Challenge goes from standard deviation 0.28 under heuristic answer matching to 0.005 under a judge. We therefore score every model three ways: heuristic string extraction, option log-likelihood, and an LLM judge. That tells us whether training reduces only answer-extraction brittleness or the underlying distributional shift that POSIX measures.

Baselines

The naive reference is the unmodified instruct model, evaluated zero-shot under every template. Next to it we report the base model, which shows the size of the instruction-tuning paradox in our setting.

The first trained baseline is multi-template fine-tuning: cross-entropy on the gold answer, with each example's template sampled from the training pool. It isolates what exposure to rewordings alone buys, with no agreement term.

The strongest competitor is single-model cross-template consistency training. It uses the same model and the same loss as above, but the "peer" is the model itself under a different template, with gradients stopped. This is exactly our objective with the two models' weights tied. It is the strongest competitor because it directly operationalizes the POSIX suggestion, costs half the memory, and any gain we claim must exceed it. It gets the same templates, the same tuning budget for $\lambda$ (an identical grid), and the same token count. No reported result was found on Lacuna for this method on any benchmark in the file. Beating it means lower held-out sensitivity on MMLU and GPQA with no loss of mean accuracy, with the difference outside seed variance across three seeds.

Two inference-time references bound the problem. SENSEL from On the Relation between Sensitivity and Accuracy in In-Context Learning abstains when answers disagree across rewordings. It reports +4.1 points of area under the F1-coverage curve over maximum probability on GPT-J-6B across ten classification datasets. We run it, together with majority voting over $K$ templates, on the untrained instruct model. Matching the voting ensemble with a single forward pass would count as success. The second reference is one-shot prompting, which POSIX shows cuts Llama-2-7b's MMLU sensitivity from 1.12 to 0.513.

Two ablations isolate our key ingredients. The first is one-way distillation from a frozen base model, which tests whether the peer needs to keep learning. The second sets $w_B \equiv 1$, which tests whether stability-gated trust matters.

Why not just…?

Why not just train one model to agree with itself across templates? This is the real threat, and the proposal is built around it. Its objective can be driven down by converging on a confident wrong answer. That is precisely the low-confidence region that ProSA and Chen et al. identify as where brittleness lives. The peer-diversity sweep settles the question. If a reseeded copy of the model matches a base or cross-family peer, the second model contributes nothing and we will report that self-consistency suffices.

Why not just add a demonstration or chain-of-thought at inference? One-shot prompting halves POSIX sensitivity for Llama-2-7b. However, Brittlebench reports that few-shot prompting can increase brittleness to formatting changes, and True Few-Shot Learning shows that choosing prompts without a large validation set carries roughly a 27% risk of a prompt 13% worse than average. Brittlebench also finds chain-of-thought reduces the accuracy drop under rewording by only about 17%. We evaluate trained models both zero-shot and one-shot to check whether the two fixes stack or substitute for each other.

Why not just ensemble rewordings at test time? It works, which is why it serves as our upper reference. Its cost is many forward passes per query, the overhead Chen et al. themselves flag. The comparison is a single-pass model against a $K$-pass vote at equal accuracy.

Why not just distill from a strong frontier teacher, or fix the evaluator? ProSA's "unstable genius" finding is that GPT-4o can be more prompt-sensitive than Llama 3-70B-Instruct on some tasks, so a strong teacher does not guarantee a stable one. Our frozen-base ablation tests one-way transfer directly. Judge-based scoring removes extraction artifacts, but it does not help when a model must emit structured output with no second model to interpret it. We report judge scores so that any remaining gain is clearly at the distribution level.

Proposed Experiments

Training uses the ARC training split and the CommonsenseQA training split, rendered through two of ProSA's four template categories (Simple Input, Role Player). The other two categories (Emotional Support, Output Requirement) are held out. Evaluation is entirely on held-out tasks and held-out perturbation families:

  • MMLU and GPQA-diamond, scored with ProSA's PromptSensiScore and POSIX $\Psi$;
  • Brittlebench perturbations (word manipulations, padding, paraphrasing), reported as $V_{brittleness}/V_{total}$;
  • the option shuffling and symbol substitution of When Benchmarks are Targets, which moved Llama2-7b's accuracy by 24.5% just by placing the correct answer at "A";
  • trailing-space and politeness edits from The Butterfly Effect of Altering Prompts.

Accuracy on BBH guards against capability loss.

The decisive checks are as follows:

  1. Peer value. Co-distillation must beat tied-weight self-consistency on held-out families.
  2. Mechanism. The size of that win must rise with measured peer error decorrelation.
  3. Failure mode. The consistent-but-wrong rate must fall rather than rise.
  4. Representation-level effect. The reduction must persist under log-likelihood and judge scoring, not only heuristic extraction.

If the first or second check fails, the proposal is refuted in its interesting form.