Letting the Student Clean Its Own Synthetic Data

Teams that cannot afford human labels increasingly have a large language model write training data for a small model. The standard recipe for search is to have the language model write a plausible query for each document and then train a compact retriever on the resulting (query, document) pairs. Many of these generated pairs are wrong: the query is off-topic, hallucinated, or too generic. Current pipelines handle this by generating in bulk and then running a separate filter that throws the bad pairs away. That approach pays twice. It pays for tokens that are later discarded, and it pays for the filter. On small or specialized collections the filter can also remove so much data that the retriever is left with too little to learn from.

Recommender systems have faced the same problem for years under a different name. Clicks are "implicit feedback": each click is treated as a positive label, yet many clicks are accidental or driven by clickbait. That field has found that a model's own training dynamics reveal which labels are wrong. Wrong labels stay hard to fit early in training, while correct ones are learned quickly. We propose to treat synthetic query–document pairs as noisy clicks. The student retriever's training trajectory would replace the separate filter. Pairs it flags would be turned into hard negatives instead of being thrown away. The same per-document signals would decide where the next unit of generation budget goes. The target is retrieval quality equal to or better than filter-based pipelines on the BEIR benchmark, at substantially fewer generated tokens and with no separate filtering model.

Core Claim

Synthetic training pairs contain errors that look statistically like false-positive clicks. Recommendation denoising supplies three reusable tools for them. The first flags pairs whose training loss stays high persistently, as opposed to pairs that are merely variable. The second calibrates that judgment per unit (per user in recommendation, per document here) instead of with one global threshold. The third relabels confirmed errors instead of deleting them. Combining these inside the student's ordinary training run makes post-hoc filtering unnecessary. Applying them early, on a small pilot pool of generations, tells us which documents deserve more generation and which do not. The new element is using the downstream student's learning dynamics to steer both how synthetic data is cleaned and how much is generated. Existing synthesis pipelines generate unconditionally and then filter with an outside model.

Where This Sits

Two synthesis pipelines motivate the problem. Promptagator prompts FLAN-137B with eight examples to write millions of queries, and discards a query if a retriever trained on the synthetic data cannot retrieve the query's source document ("round-trip" filtering). It reports that filtering removed critical data on NFCorpus. DISCO generates counterfactual sentences with GPT-3 and keeps only those a DeBERTa teacher confirms. Both show that raw generations are noisy and that filtering helps. Both also treat the filter as an external stage that has no link to how the student learns, and both delete what they reject. Cost analyses of API-based synthesis (Shi et al., LMGQS, Wozniak et al.) and the saturation result in Beyond Quantity support the premise that generation volume is the dominant and poorly spent cost.

Recommendation denoising supplies the method. Adaptive Denoising Training drops or down-weights high-loss interactions early in training, which exploits the fact that networks memorize noise late. We reuse its core observation, but it uses a global threshold and discards data. Personalized Loss Denoising judges each interaction against the user's own loss distribution, so valid niche preferences are not removed. We transfer this as per-document calibration, since documents differ greatly in how well a generator can write queries for them. The Double Correction Framework averages loss over a sliding window and uses a concentration-inequality lower bound to separate persistent noise from high-variance hard samples. It then relabels confirmed noise from positive to negative to avoid starving the model of data. This is the closest method to ours. We apply it to synthetic pairs, where "relabel to negative" becomes "use the hallucinated query as a hard negative for its source document." We also feed its statistics back into generation, which DCF never does because clicks are not generated on demand.

The diffusion methods in recommendation, ADAR and DDRM, do connect synthesis with feedback noise. They synthesize negative embeddings or denoise representations, but they add multi-step sampling cost and never touch text generation. They constrain our design: we deliberately add no generative model beyond the language model already in the pipeline. Counter-IF is relevant to the false-negative side, because other documents may also answer a synthetic query. We treat that side only as an ablation.

Research Program

The pipeline has three stages that share one student training run.

In the pilot stage, an open instruction-tuned language model, prompted with Promptagator-style few-shot examples, writes a small number of queries per document, for example two. The student is a dual encoder of about 110M parameters, a retriever that embeds queries and documents separately and scores them by similarity. It trains on these pairs with in-batch negatives. Throughout training we log each pair's loss.

In the diagnosis stage, we compute DCF's confirmed loss for each pair: its windowed mean loss, together with a lower confidence bound that accounts for its variance. We compare this against the distribution of confirmed losses for pairs from the same document, and also against the global distribution, with a mixing weight chosen on a held-out split. A pair is flagged when its lower bound exceeds the calibrated threshold. This means its loss is high persistently, not just noisily. Flagged pairs are not deleted. Each flagged query becomes a hard negative for its source document, down-weighted by the flag's confidence, and the student continues training.

In the allocation stage, per-document statistics from the pilot set the remaining generation budget. Some documents have mostly unflagged pairs that remain unfit, meaning they are learnable but hard; these receive more queries. Some are already easy; these receive none. Some have mostly flagged pairs, which suggests the generator fails on them; these receive a capped amount with a changed prompt instead of more of the same. We run one or two such rounds, reusing the same student and continuing its training rather than retraining.

A side analysis measures the flag precision directly. We audit a stratified sample of flagged and unflagged pairs for relevance, using human judgments on a subset and a stronger language-model judge on the rest, and compare the flags with Promptagator's round-trip decisions on the same pairs.

Baselines

The naive reference is training the student on all raw generations with no filtering, at the same total generation budget. It shows how much noise actually hurts at this scale. We will also report BM25 zero-shot, because BEIR found it a strong out-of-domain baseline that beats several dense retrievers. The file has no specific BM25 number, so we will measure it ourselves on each dataset.

The strongest existing method is Promptagator's round-trip filter. Its paper reports 47.8 average nDCG@10 over 11 BEIR datasets with eight examples, versus 45.5 zero-shot, and 52.8 with a reranker. That result used FLAN-137B, which we cannot run. We will therefore reimplement the filter with the same open generator, the same raw query pool, the same student architecture, and the same number of training steps and hyperparameter search budget as our method. We also count the GPU-hours of its auxiliary filter retriever against it. Beating it means equal or higher average nDCG@10 at a matched token budget, or equal nDCG@10 at clearly fewer generated tokens, with no loss on NFCorpus-type small collections. We do not expect to reproduce 47.8 in absolute terms; the comparison is internal.

A second strong competitor is plain ADT truncated loss applied to the synthetic pairs: drop the highest-loss fraction on a rising schedule. It is the cheapest trajectory-based method.

Three ablations isolate our ingredients. The first uses a global threshold instead of a per-document one. The second deletes flagged pairs instead of recycling them. The third uses uniform generation allocation instead of trajectory-guided allocation, at the same total token budget.

Why not just…?

Why not just keep Promptagator's round-trip filter? It works, and it is our main baseline. But it deletes rejected queries, and its authors report data starvation on NFCorpus. It also trains a separate retriever whose only job is filtering. It gives no signal about where more generation would help. The token-budget curves and the NFCorpus results will show whether our approach recovers what the filter throws away. If round-trip filtering matches us at every budget, then the contribution reduces to cost accounting, and we will report it as such.

Why not just apply ADT's truncated loss directly? This is the real threat, because it is nearly free and already uses the student's trajectory. PLD and DCF both show that global high-loss rules confuse hard or niche samples with noise, and DCF shows that dropping causes sparsity. Those findings come from click data, though, not synthetic queries. The global-threshold and delete-instead-of-recycle ablations settle this. If truncated loss comes within noise of the full method, we will adopt it as the cleaning step and keep only the allocation contribution.

Why not just generate fewer queries, or scale up raw generation instead of cleaning? DISCO found that 75,000 filtered examples beat over a million noisier ones, and Beyond Quantity reports early saturation, so volume alone is unlikely to win. Uniform down-scaling, however, is exactly the allocation ablation at lower budgets. If uniform allocation traces the same nDCG-versus-tokens curve as guided allocation, the allocation idea fails.

Proposed Experiments

We evaluate on the BEIR datasets used by Promptagator, including ArguAna, Touché-2020, FEVER, Climate-FEVER, HotpotQA, SciFact, and NFCorpus. Metrics are nDCG@10, generated tokens, total GPU-hours including any filter, and the fraction of generated tokens that end up unused. The central plot is nDCG@10 against generated tokens for each method, swept over four budgets.

We will run four decisive checks. First, at matched budget, the full method should at least match the round-trip filter on average and exceed it on NFCorpus and SciFact, where starvation is most likely. Second, guided allocation should reach the uniform-allocation plateau with meaningfully fewer tokens; we will treat a saving below 25% as a negative result. Third, the audit should show that trajectory flags are at least as precise as round-trip rejections. It should also show that per-document calibration spares more truly relevant but hard queries than the global threshold does. Fourth, recycled hard negatives should not hurt. If they lower nDCG@10, the "relabel rather than delete" lesson from DCF does not transfer, because a hallucinated query can still be lexically close to relevant text. We will report that as a finding that constrains how the two literatures connect.