Held-Out Teachers as Auditors for Label-Free Prompt Optimization

Large language models are increasingly improved by rewriting their prompts instead of retraining their weights. A second model reads the task model's mistakes, writes a critique in plain language (a "textual gradient"), and edits the prompt in response. On unlabeled data there are no gold answers, so the mistakes have to be defined against guessed answers, called pseudo-labels, and wrong guesses lead to wrong critiques. The current label-free method in this line generates those guesses from the very model being optimized, using majority vote over its own samples. When that model has a blind spot, every sample repeats the same error, and the optimizer then edits the prompt toward that error.

We propose to distill a committee of different teacher models into a small student's prompt, and to split the committee's jobs. Some teachers supply the pseudo-labels from which critiques are written. A separate, held-out group of teachers decides whether each resulting prompt edit is kept. An edit survives only if it moves the student toward a majority of the held-out teachers, each judged on its own. This borrows the logic of tri-training, where two models label data for a third, and of gradient-orientation voting from multi-teacher distillation. Here it acts on discrete prompt edits rather than on labels or numerical gradients. The claim is that in label-free prompt optimization, the costly noise is not only wrong pseudo-labels. It is also edits that fit the biases of whichever teachers produced the labels, and only an auditor independent of the label source can catch those.

Core Claim and Positioning

The nearest existing method is PAPO. PAPO estimates confidence by self-consistency on one base model, puts high-confidence self-labeled examples into the prompt as demonstrations, and then runs TextGrad on the remaining errors. TextGrad is the general system that turns LLM critiques into prompt updates through a PyTorch-like backward-and-step interface. PAPO reports GPQA going from 37.9% to 39.9%. Our proposal differs in two ways. First, the pseudo-labels come from architecturally different teachers rather than from the student itself. Second, and more important, edit acceptance is decoupled from label production.

The obvious version of this project would simply swap PAPO's self-consistency for a teacher majority vote. That version is a baseline here, not the contribution. A skeptical reviewer would rightly expect it to capture much of the benefit of having several teachers.

Two other lines constrain the design. The first is work on multi-agent prompt optimizers. MAPGD fuses critiques from several role-prompted agents by clustering their embeddings. MARS runs a Socratic teacher–critic–student dialogue. Both assume gold labels, and both treat every critique as valid, differing only in which part of the prompt it addresses. Neither can tell when a critique rests on a false premise. We reuse their insight that conflicting critiques must not be averaged, but we filter edits by outcome rather than fusing critiques by content.

The second line is multi-teacher distillation, which supplies our method. GOVERN updates a student only along the direction most teachers vote for, instead of their averaged logits. It reports 76.43% on qp R@P=90% versus 74.63% for mean-ensemble distillation. MACPO and Learning from All treat cross-teacher consensus as a positive preference signal and individual deviations as negatives. All three update continuous weights. We carry the voting principle over to discrete edits of a frozen student's prompt.

The pseudo-label noise literature motivates the decoupling itself. Tri-net accepts a pseudo-label for one head only when the other two agree. Auto-Annotation Quality Prediction shows that disagreement within an ensemble catches errors that raw confidence misses: the rare-class "train" IoU rose from 55.2% to 74.9%. MACPO reports that self-generated supervision peaks early and then collapses. SkillProx warns that critiques can be "semantically plausible" yet "functionally incorrect", and that unaudited edits bloat prompts. Its own audit requires verified tasks. We supply a label-free substitute for that audit.

Research Program

The setting has three parts. The student is a small, frozen open model whose system prompt is being optimized. The committee is a set of K heterogeneous teacher models from different families, for example the Llama, Mistral and Qwen families used across the cited work. The data are unlabeled task inputs.

At each round, the committee is randomly partitioned into a source group A and an audit group B. Teachers in A answer a batch of inputs. Their majority answer, where they agree, becomes the pseudo-label, and the student's mismatches against those labels feed a standard TextGrad critique-and-edit step that proposes candidate prompts. Each candidate prompt p' is then scored on a fresh unlabeled audit batch U against each audit teacher t in B separately.

$$\Delta_t(p') = \mathrm{agree}_U(\text{student}_{p'}, t) - \mathrm{agree}_U(\text{student}_{p}, t)$$

Here agree_U is the fraction of items in U on which the student, run with a given prompt, gives the same answer as teacher t, and p is the current prompt. The candidate is accepted only if more than half of the teachers in B have Δ_t > 0. This is an orientation vote in GOVERN's sense: it asks whether most auditors agree the edit moves the student toward them. It does not use averaged agreement, so a single idiosyncratic auditor cannot carry the decision.

The same audit drives a pruning pass in the style of SkillProx. Every few rounds, each sentence or demonstration in the prompt is removed one at a time. A component is deleted if the vote does not object to its removal. This is meant to control prompt growth without gold labels.

Rotating the partition across rounds means every teacher both produces and audits signal over the run, and no teacher ever audits an edit derived from its own labels.

The analysis will separate three quantities, using gold labels that are hidden from the optimizer and used only for measurement:

  • the error rate of the pseudo-labels;
  • the "harmful acceptance rate", meaning the fraction of accepted edits that lower gold accuracy;
  • the length of the final prompt.

Our hypothesis concerns where the gain comes from. Switching from self-labels to committee labels should mainly reduce pseudo-label error. Decoupling the auditors should mainly reduce harmful acceptances, and it should help most when the source teachers share a bias.

Baselines

The naive reference is the student run with its initial hand-written prompt and no optimization. Next to it, we will report the teacher committee's own majority-vote accuracy at inference time. That figure is a cost-unbounded upper reference showing what the committee knows. The deployment argument is that the optimized student matches part of it at a fraction of the inference cost.

The strongest existing label-free method is PAPO, re-run on our student with its single-model self-consistency labels. The cited reference point is the GPQA result of 37.9% to 39.9% reported in PAPO. We will also re-run PDO, which ranks candidate prompts with an LLM judge through pairwise duels on unlabeled data and reports the best accuracy on 13 of 16 BBH tasks. The file gives no per-task scalar for PDO, so we treat it only as a re-run competitor. These two are the strongest competitors because they are the only methods in the file that optimize prompts without labels.

The most dangerous baseline is committee-majority pseudo-labels feeding vanilla TextGrad, with edits accepted by accuracy against those same pseudo-labels on a validation split. This is the "obvious idea", and it uses exactly the same teachers we do. A MAPGD-style variant, which fuses critiques from all teachers by clustering instead of filtering edits, tests whether combining critiques by content is a substitute for auditing outcomes.

All methods receive the same unlabeled pool, the same student, the same number of optimizer rounds, and the same total budget of LLM tokens. That budget counts teacher queries. It is the fair currency because our audit spends teacher calls that the baselines can spend on more critique steps instead.

As a ceiling, we include gold-label TextGrad. For context we cite MAPGD's supervised GSM8K result of 93.5% versus a 90.0% baseline, though that figure comes from a different model setup and is not directly comparable.

Two ablations isolate the key ingredient. The first lets the same teachers both label and audit, which removes the decoupling. The second replaces the orientation vote with mean agreement. A third ablation drops pruning.

Beating the committee-majority TextGrad baseline by a margin larger than seed variance is required, across three or more seeds on at least two of the task families. Showing a lower harmful acceptance rate under the same budget is also required. Matching PAPO alone does not count as success.

Why not just…?

Why not just take the committee's majority vote as pseudo-labels and run TextGrad? This is the real threat, and it may capture most of the gain on tasks where the teachers' errors are close to independent. Our hypothesis is narrower. We expect the extra benefit of a separate audit to appear when the source teachers share errors, which happens often among models trained on overlapping data. In that case, validating against their own majority approves edits that encode the shared mistake. The same-teacher ablation and the correlated-committee stress test below settle this directly. If decoupling adds nothing even under deliberately correlated committees, we will report that the simple version suffices.

Why not just use self-consistency on one strong model? That is PAPO's design. The file records that it struggles when the base model cannot produce correct answers to begin with, in PAPO itself. MACPO reports that self-generated supervision such as SPIN and self-rewarding peaks early and then degrades. We include PAPO directly and track accuracy across rounds to test for that late collapse.

Why not just fine-tune the student on committee-consensus labels? When the student's weights are accessible and the budget allows training, this is a strong option. MACPO and Learning from All suggest that weight-level preference distillation from multiple teachers works well. Our target is a frozen or API-served student, or a setting where only a few hundred unlabeled items exist. On open students we will run a LoRA fine-tune on the same consensus labels as a reference. If it dominates at equal data, the prompt-level method's case reduces to deployability and interpretability, and we will say so.

Why not just ensemble the teachers at test time? It will usually be more accurate. It also multiplies inference cost by K, which is the motivation GOVERN gives for distillation. We report it as an upper reference and frame the result as the accuracy-per-cost trade-off between the two.

Proposed Experiments and Decisive Checks

Tasks are drawn from the file's benchmarks so that they span the regimes where pseudo-labels behave differently:

  • BIG-bench Hard tasks used by PDO and MARS, including Geometric Shapes, where PDO's single judge is known to struggle.
  • GSM8K, where answers are exact and committee agreement is informative.
  • GPQA and MMLU College Math, where teacher accuracy is low and correlated errors are likely.
  • RTE, as a near-ceiling control.

Metrics are gold accuracy of the optimized student, harmful acceptance rate, pseudo-label error, final prompt length, and total tokens consumed.

The first decisive check is a correlated-committee stress test. We will build committees whose source group contains two models from the same family alongside one dissimilar model, and compare the full method against the same-teacher-audit ablation. The hypothesis predicts that decoupling matters most here and least for maximally diverse committees.

The second check measures how prompt length and gold accuracy evolve over rounds, with and without vote-based pruning. It tests whether the label-free audit reproduces SkillProx's finding that pruning helps, in a setting where SkillProx's verified audit is unavailable.

The third check is an out-of-distribution transfer test. Prompts optimized on one BBH task family are evaluated on held-out families. This asks whether audited edits are less tied to the specific source teachers than unaudited ones.

If the full method's advantage over committee-majority TextGrad vanishes in all three checks, the conclusion will be that multiple teachers matter only as a source of labels, not as auditors. That would be a useful negative result for anyone building label-free prompt optimizers.