Many retrieval systems are now trained on synthetic data. A large language model reads each document in a target collection and writes a plausible search query for it. A small, fast retriever is then trained to match those queries back to their documents. Because the model's queries are often off-topic or invented, pipelines add a "round-trip" check: a temporary retriever looks up each generated query, and if the source document does not come back at the top, the query is thrown away. This design has two costs. Every discarded query is paid-for generation that produces nothing. And the keep-or-discard decision is a single hard call made by a weak model, so on small, specialized collections it can reject good queries and leave the retriever with too little training data.
We propose to treat the round-trip check as an uncertainty signal that steers generation, not as a gate that follows it. For each document, the model first writes a small ranked list of candidate queries. Each candidate is kept and weighted by how well the round-trip retriever recovers its source document. Further generation is requested only for documents where no candidate is well grounded. The question is whether this gives a better retriever for the same number of generated tokens, or the same retriever for fewer, compared with uniform over-generation followed by a filter. This combines two ideas from older pipeline research: keep several ranked guesses alive instead of committing early, and widen the search only where the upstream stage is unsure. Here they are applied to the stage that dominates synthetic-data cost, which is calling the generator.
Core Claim
Under a fixed budget of generator tokens, adaptive per-document allocation of candidate queries, combined with soft round-trip weights, will yield a stronger student dual encoder than uniform generation with a tuned hard filter. A dual encoder is a retriever that embeds queries and documents separately so search is a fast nearest-neighbor lookup. We expect the gain to be largest on small corpora where hard filtering starves training. The claim is about the shape of the cost–quality curve, not a single operating point. It is falsifiable: if a uniform budget with a well-tuned acceptance threshold traces the same curve, the idea is not worth its complexity.
Grounding in Prior Work
The synthetic-data side supplies the setting and the motivating failure. Promptagator prompts FLAN 137B with eight examples to write queries over BEIR corpora. It filters them with a round-trip check using a retriever trained on the raw synthetic data, and trains a 110M dual encoder. It reports 47.8 nDCG@10 averaged over 11 BEIR datasets, against 45.5 for zero-shot generation, and 52.8 with an added reranker. It also notes the filter was overly strict on NFCorpus. We keep its recipe but replace the filter-after-generation order.
UDAPDR avoids hard filtering in a different way. It generates up to a million queries with Flan-T5 XXL, trains several DeBERTa-v3-Large rerankers on them, and distills their scores into ColBERTv2, lifting LoTTE Lifestyle Success@5 from 64.5 to 74.8. It shows soft supervision helps, but it moves the cost into training multiple cross-encoders and never changes how much is generated per document. Our method keeps generation as the lever and adds no teacher models.
DISCO constrains the design. Its 75k filtered counterfactuals beat more than 1M noisy ones, so we should not expect "keep everything" to work. Soft weights must still push down bad candidates rather than simply restore them.
The pipeline side supplies the method. Beyond myopic inference in big data pipelines passes top-k hypotheses between stages, gaining about 3% MAP on ACE 2004 from Top-2 over Top-1. It shows that adaptive inference, which widens the beam only when a stage is uncertain, recovers most of the benefit of full propagation at a fraction of the search. Its stages were classical taggers and parsers, and its cost was database joins. We move the adaptive rule to the point where cost is generator calls.
How to loose confidence shows that softmax-style scores are overconfident and collapse candidate sets back to one choice. This is why we build grounding weights from the source document's rank, not from normalized retriever probabilities.
Modeling Input Uncertainty in Neural Network Dependency Parsing shows a probability-weighted mix of top-N candidates beats 1-best input and approaches gold. Our weighting of multiple queries per document in the training loss is the discrete analogue.
Research Program
The generator is Flan-T5 XXL, following UDAPDR, so every condition is reproducible and the token cost is measurable. It is prompted with Promptagator-style eight-example prompts for each corpus.
For each document $d$, a first call returns $n_0$ candidate queries. We try both beam-search n-best with sequence log-probabilities and independent sampling, since beam lists are often near-duplicates. The round-trip retriever, trained as in Promptagator on an initial raw pass, retrieves for each candidate $q$. We record the rank $r(q,d)$ of the source document and assign the grounding weight
$$w(q,d) = \frac{1}{r(q,d)} \cdot \mathbb{1}[r(q,d) \le R],$$
where $R$ is a rank cutoff beyond which a candidate is treated as ungrounded. The document's grounding level is $g(d) = \max_q w(q,d)$.
Documents with $g(d)$ below a threshold $\tau$ receive further generation rounds. Each round uses a different few-shot prompt and stops once $g(d) \ge \tau$ or a per-document cap is reached. The total token budget $B$ is fixed across conditions. The student dual encoder is trained with a contrastive loss in which each (query, document) pair is scaled by $w$. We also report the total compute spent on round-trip retrieval, because the adaptive loop calls it more than once.
The analysis goes beyond headline scores. We record which documents consume extra budget, whether their extra candidates end up grounded, and how the share of generated tokens that survive with non-trivial weight changes compared with the uniform pipeline. The last measure is a direct count of discard waste.
Baselines
The naive reference points are BM25 and a dual encoder trained on no target-domain data. BEIR shows BM25 often beats zero-shot dense models under domain shift, so it is the "do nothing" floor. A synthetic-data method that cannot beat it on small corpora has failed.
The main published competitor is the Promptagator recipe. Its reported 47.8 nDCG@10 average over 11 BEIR datasets used FLAN 137B and cannot be matched with our generator. We therefore reimplement the recipe with Flan-T5 XXL: identical prompts, round-trip retriever, student architecture, and training steps. We sweep uniform queries-per-document $K$ so its cost–quality curve covers our budget range. No reported per-dataset Promptagator result for NFCorpus alone was found on Lacuna, so the per-corpus comparison rests on our reimplementation.
The strongest fair competitor is the same uniform pipeline with its acceptance rule tuned on a held-out development split. It accepts a query if the source document is within top-$k$ rather than top-1, and it receives the same tuning budget as our $\tau$ and $R$. This is the competitor most likely to erase our gain.
UDAPDR-style distillation is a heavier reference. We train one DeBERTa-v3-Large reranker on the uniform synthetic set and distill its scores into the student. Its training compute is reported alongside nDCG, so readers see quality per unit cost rather than quality alone. Its published LoTTE Lifestyle Success@5 of 74.8 is the reference number for that domain.
Three ablations isolate the key ingredient. Soft weights with uniform allocation tests weighting alone. Adaptive allocation with hard filtering tests allocation alone. Random allocation of the same extra budget, and reversed allocation that gives extra rounds to already-grounded documents, test whether uncertainty-directed spending matters or whether any extra spending helps.
Beating the tuned-threshold baseline means one of two things across budget levels. The first is at least a 1-point nDCG@10 average gain over the small BEIR corpora at matched tokens. The second is matching its best nDCG@10 with at least 30% fewer generator tokens.
Why not just…?
Why not just loosen the filter threshold? This is the most serious threat. Accepting a query when its source document is in the top-5 already softens the hard decision, and it may recover most of what NFCorpus lost. A loosened threshold still spends the same generation on every document, including those whose first query was clearly fine, so it cannot reduce waste. It is also still binary between accept and reject. The tuned-threshold baseline and the "soft weights, uniform allocation" ablation settle this directly. If both match the full method at every budget, we will report that adaptive allocation adds nothing.
Why not just generate more queries per document? Uniform over-generation is the default. DISCO's result that 75k filtered examples beat more than 1M noisy ones suggests volume alone is not the lever. Uniform over-generation spends most extra tokens on documents that were already covered. The uniform-$K$ sweep traces exactly this curve, and the claim is that ours lies above it.
Why not just distill a reranker's soft scores, as UDAPDR does? UDAPDR shows soft teacher supervision works. It cuts nothing on the generation side, and it adds cross-encoder training that is large relative to the student. The two ideas can also be combined. We report the single-reranker variant with its compute and test adaptive generation feeding reranker distillation, to check whether the gains stack or overlap.
Why not just use temperature sampling instead of an n-best list? Independent samples may be more diverse than a beam, and diversity may matter more than log-probabilities. We treat this as an open design choice, not a strawman. Both are run at matched tokens inside the adaptive loop.
Proposed Experiments
The evaluation uses the smaller BEIR corpora where filtering starvation should appear: NFCorpus, SciFact, ArguAna, and Touché-2020. FEVER and HotpotQA serve as large-corpus controls where we expect little gain. LoTTE's five domains test the long-tail setting UDAPDR studies. The metrics are nDCG@10 for BEIR and Success@5 for LoTTE, each plotted against generator tokens, round-trip retrieval calls, and total GPU hours, over three seeds per point.
The decisive checks are four. At matched generator tokens, the full method must beat the tuned-threshold uniform baseline by the margin stated above on the small corpora. Random and reversed allocation must fall clearly below uncertainty-directed allocation. The fraction of generated tokens that survive with non-trivial weight must rise substantially over the uniform pipeline. The gain should shrink on large corpora, where hard filtering does not starve training. If the adaptive curve coincides with the tuned-threshold curve, the conclusion is that simple threshold tuning is sufficient, and that is a useful negative result for anyone paying for synthetic data.
References
- Promptagator: Few-shot Dense Retrieval From 8 Examples
- UDAPDR: Unsupervised Domain Adaptation via LLM Prompting and Distillation of Rerankers
- DISCO: Distilling Counterfactuals with Large Language Models
- Beyond myopic inference in big data pipelines
- How to loose confidence: probabilistic linear machines for multiclass classification
- Modeling Input Uncertainty in Neural Network Dependency Parsing
- BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models