Spending the Teacher Only Where the Student Needs It: Budgeted, Closed-Loop Pseudo-Labeling for Retriever Adaptation

A search engine trained on general web queries usually performs poorly on a specialized collection such as financial forums or biomedical abstracts. A common fix needs no human labels. A text generator writes synthetic questions for each target document. A large, slow scoring model (the "teacher") then grades how well each document answers each question. Finally, a small, fast retriever (the "student") is trained to copy those grades, which is a form of knowledge distillation. The expensive step is the teacher. It is run once, over everything, before the student has learned anything. This note proposes running it in rounds. After each round, the partly trained student decides which documents are worth sending to the teacher next. It skips documents it already handles well, and it also skips the documents it finds hardest, because those are exactly where the teacher is most likely to be wrong.

The claim is that this closed loop can match single-pass adaptation using a small fraction of the teacher's compute. The loop also lets us track something single-pass pipelines never measure: how much general-purpose retrieval ability the student loses as it specializes, and whether choosing documents carefully reduces that loss. If a random subset of the same size does just as well, the more important finding is that existing pipelines oversample. The experiments are designed so that this outcome would be clearly visible.

Core Claim

GPL is the method this proposal builds on, and it is the strongest baseline. It generates queries for target passages with T5, mines hard negatives with a dense retriever, scores query–passage pairs with a cross-encoder (a model that reads query and passage together and outputs a relevance score), and trains a bi-encoder (a model that embeds queries and passages separately so search is fast) to match the cross-encoder's score margins. It reports gains of up to 9.3 nDCG@10 over MS MARCO-trained bi-encoders on BEIR. It also names the offline cross-encoder cost as an open problem. UDAPDR pushes the same one-directional design further: GPT-3 seed queries, one million Flan-T5 queries, several rerankers, then distillation into ColBERTv2. It raises LoTTE Lifestyle Success@5 from 64.5 to 74.8. In both pipelines the student never influences what the teacher labels, and neither measures what the student forgets.

Our method supplies that missing influence. In round $t$, the current student $f_t$ scores each unlabeled target passage $p$ against a query $q$ produced by the cheap generator, and against the hard negatives it retrieves itself. From these scores we compute a margin:

$$m_t(q,p) = s_t(q,p) - \max_{p^- \in N_t(q)} s_t(q,p^-)$$

Here $s_t$ is the student's similarity score and $N_t(q)$ is its top retrieved non-source passages. Pairs with a large margin are already solved, so we skip them. Pairs with a very negative margin are often bad queries or ambiguous passages. Robust Active Distillation shows, in its "hardness trap" result, that the instances a student is least sure about are the ones a teacher most often mislabels. That paper therefore samples moderately uncertain instances, with a caution level set by the estimated teacher error rate. We take this selection rule and move it to generative pseudo-labeling under domain shift, a setting the paper does not study. Only pairs in the middle margin band go to the cross-encoder. Each round, negatives are re-mined with the updated student.

The cross-encoder is the source of labels throughout. The student never labels itself. This matters because WILDS 2.0 found that classic self-training, where the model trusts its own confident predictions, fails to beat plain training on text shifts. The iteration in our loop only changes which pairs the teacher sees. It never changes whose labels count.

Research Program

The build is a loop wrapped around the GPL codebase design. Its accounting unit is teacher cost: cross-encoder forward passes plus generator tokens, and LLM tokens in an LLM-generator variant. Every method receives the same total teacher budget, split into four to six rounds.

The analysis has three parts. The first is a cost–quality curve, plotting target nDCG@10 against teacher passes. The second is a retention curve, plotting nDCG@10 on MS MARCO dev and on BEIR datasets outside the target against the same axis. The third is a label-noise audit. On the small set of target queries that have human relevance judgments, we check whether the moderate band really has lower cross-encoder error than the lowest-margin band. That is the hardness-trap claim, re-tested under domain shift.

The specialization question is motivated by evidence that narrow tuning of small models erodes general skill. Specializing Smaller Language Models towards Multi-Step Reasoning distills math chain-of-thought into Flan-T5 and sees BigBench Hard ability collapse. ClinicBench finds that clinical instruction tuning harms summarization. Retrievers are a different setting, and we do not assume the same collapse. We measure it. Our hypothesis is that noisy low-margin pairs distort the embedding space more than clean ones do, so careful selection should forget less at equal target gain. This is a secondary hypothesis. The main claim stands or falls on cost.

Our approach also relates to LLM-enhanced Self-training for Cross-domain Constituency Parsing. That work iterates targeted LLM generation for parsing and shows that 10,000 targeted sentences per round beat 40,000 crawled ones. It supports the idea that targeting beats volume. However, it pays full generation cost every round and does no distillation. Our method differs in two ways. The student itself chooses where to spend the teacher. And the cheap generator runs broadly, while only the expensive scorer is rationed.

Baselines

The naive reference point is the MS MARCO-trained bi-encoder applied zero-shot. For context, Promptagator reports 45.5 nDCG@10 averaged over its BEIR set for zero-shot prompting and 47.8 for 8-shot prompting, using a 137B teacher. We do not reproduce that teacher. We cite these numbers only to show the scale of gains that heavy generation buys.

The strongest competitor is full single-pass GPL, run on every target passage with an unlimited teacher budget. It is the published method the proposal is meant to make cheaper, so it sets the quality ceiling. The file gives GPL only an aggregate "up to 9.3 nDCG@10" gain. There is no reported result on Lacuna for GPL's per-dataset nDCG@10 on FiQA, SciFact, BioASQ, or NFCorpus, so we will rerun it with the same base bi-encoder, cross-encoder, generator, and training steps. Beating it means matching its target nDCG@10 within 0.5 points while using at most 30% of its cross-encoder passes, or exceeding it at equal budget.

The critical fair baseline is budget-matched GPL. It is single-pass GPL on a uniformly random subset of passages sized to our total teacher budget, with identical hyperparameters and the same number of student gradient steps.

Two ablations isolate the key ingredient. The first, iterative-random, keeps our rounds and negative re-mining but selects passages at random, which separates the benefit of iteration from the benefit of selection. The second, iterative-hardest, selects the lowest-margin pairs, which is ordinary active learning and tests the hardness trap directly.

On LoTTE we also report UDAPDR's published 74.8 Success@5 on Lifestyle as an external reference. That number comes from a much larger generation budget, and we present it as such rather than as a matched comparison.

Tuning is identical for all methods. Hyperparameters are chosen on one development domain, FiQA, and frozen for the others, because target labels are unavailable in the intended use.

Why not just…?

Why not just run GPL once on a random subset of the same size? This is the real threat. If synthetic pairs are highly redundant, random subsampling alone may recover most of GPL's gain, and the student adds nothing. The budget-matched GPL baseline tests this directly, across budgets from 5% to 50% of full GPL. If our method's curve does not sit clearly above the random curve at low budgets, we report that single-pass pipelines are oversampled and that student guidance is unnecessary. That would still be a useful and inexpensive finding.

Why not just pick the passages the student is least sure about? That is standard active learning, but Robust Active Distillation shows this is where teachers err most, so maximum-uncertainty sampling feeds the student the noisiest labels. The iterative-hardest ablation and the label-noise audit test whether that result holds under domain shift with a cross-encoder teacher.

Why not let the student pseudo-label itself, or just continue pretraining on target text? WILDS 2.0 reports that pseudo-labeling, FixMatch, and masked-language-model pretraining gave no significant gain over plain training on CivilComments and Amazon. GPL reports that TSDAE pretraining alone was insufficient without cross-encoder pseudo-labels. We keep an external teacher for exactly this reason, and we include a TSDAE-then-our-loop variant so that any complementary benefit shows up.

Why not just mix MS MARCO replay into training to prevent forgetting? Replay is cheap and may remove most of the specialization tax regardless of selection. We therefore run every method with and without a fixed replay ratio. The secondary hypothesis survives only if careful selection reduces forgetting beyond what replay already achieves.

Proposed Experiments

Target domains are BEIR's FiQA, SciFact, BioASQ, and NFCorpus, plus LoTTE Lifestyle and Technology. Retention is measured on MS MARCO dev and on the BEIR datasets not currently being adapted to. The primary metric is target nDCG@10 (Success@5 on LoTTE) as a function of cross-encoder passes. Secondary metrics are the retention drop relative to the zero-shot student and cross-encoder error by margin band on labeled queries.

Three results would each be decisive in their own direction. The first is a clear gap between our method and budget-matched GPL at 10–20% budget on at least four of six domains. The second is iterative-hardest underperforming the moderate band, together with measurably higher teacher error in the hardest band. The third is lower retention loss than iterative-random at equal target gain, with and without replay. If the first fails, the proposal reduces to a cost study of GPL. If the second fails, the hardness-trap transfer does not hold for retrieval. If the third fails, selection is a cost tool and not a way to protect general ability.