Training a search model on queries written by a language model has become a standard way to adapt retrieval to a new domain that has no labeled data. Promptagator prompts a 137B model with eight examples, and UDAPDR generates a million queries. Most of the cost sits in the generation step, and much of the output is wasted. DUQGen matches million-pair pipelines with 1,000 well-chosen pairs. The natural next step is a closed loop: let the retriever's current state decide which documents get more queries, and stop when extra queries stop helping. We argue that the obvious version of this loop, which generates more for documents where the retriever's loss is high, is the wrong design. The reason comes from a separate literature on training generator/encoder pairs.
The key observation is that a retriever trained on generated queries is an inference network: a model that maps an observation (the query) back to the hidden cause that produced it (the source document). The language model plays the generative model, mapping documents to queries. Work on decoupled generative-inference training shows two things. First, the inference network's learning is limited by the signal-to-noise ratio of its gradients, not by how large its loss is (Tighter Variational Bounds are Not Necessarily Better). Second, when the two sides steer each other through one shared signal, they drift into degenerate joint solutions (Characterizing and Avoiding Problematic Global Optima of Variational Autoencoders). We propose a generation controller built on both lessons. It allocates the generation budget by per-cluster gradient signal-to-noise ratio. It judges query validity with a frozen scorer that never sees the retriever's updates.
Core Claim
Consider an adaptive synthetic-data loop that steers generation with the retriever's own loss. We claim it inherits the co-adaptation failure of jointly trained encoder-decoder pairs. High loss concentrates on queries that are ambiguous, unanswerable, or hallucinated, so the loop pours budget into noise. Round-trip consistency filtering is also coupled: it checks queries with a retriever trained on the same synthetic data. A decoupled controller should do better at every budget, and it should reach the accuracy of full-corpus unfiltered generation with roughly an order of magnitude fewer generated queries. That controller has two parts. The first measures, for each document cluster, how consistently freshly generated queries push the retriever's parameters in the same direction. The second weights queries by a fixed, retriever-independent validity score.
The controller's allocation signal for a cluster $c$ of documents is a gradient signal-to-noise ratio:
$$\mathrm{SNR}_c = \frac{\lVert \bar g_c \rVert}{\sqrt{\operatorname{tr}\,\widehat{\mathrm{Var}}(g_c)/n_c}}.$$
Here $g_c$ are the retriever's loss gradients, projected onto a small random subspace of its final-layer parameters, for the $n_c$ probe queries generated so far for that cluster. $\bar g_c$ is their mean and $\widehat{\mathrm{Var}}$ their sample variance. A high value means new queries from this cluster agree on a useful update. A low value has two possible causes: the retriever has saturated the cluster, which shows up as a small mean, or the generator's queries there disagree, which shows up as high variance. In neither case should more budget go there. This is the retrieval analogue of Rainforth et al.'s result that encoder progress is governed by gradient SNR, which can collapse even as the estimate improves.
Research Program
The pipeline starts from the Study on LLMs for Promptagator-Style Dense Retriever Training (Promptodile) setup. It uses the same few-shot prompts, Llama-3.2-3B as the generator, and unsupervised E5 Base as the starting retriever. It changes only how generation calls are spent. The target corpus is clustered once, following DUQGen. Each cluster gets a small seed allocation. Training then alternates between two steps. In the first, a few probe queries per cluster are used to estimate $\mathrm{SNR}_c$. In the second, the next block of generation budget is spent in proportion to each cluster's estimated SNR, and the retriever is updated on the new queries. Generation stops globally when the budget-weighted SNR across clusters falls below a noise floor. That floor is calibrated on a single held-out domain and then fixed for all others.
Query validity is handled by a frozen scorer, which is the decoupling step borrowed from Variational Inference and Model Selection with Generalized Evidence Bounds. That paper trains the generator and inference sides with different estimators so that neither corrupts the other's signal. Our scorer is the generator's own log-likelihood of the query given its source document, minus its log-likelihood given a random corpus document. This costs one parallel forward pass per pair rather than autoregressive decoding. The difference becomes a clipped sample weight in the retriever's contrastive loss, replacing hard filtering. The scorer is never updated from retriever feedback, so the loop cannot learn to certify its own mistakes.
Two distinctions separate this design from nearby work. TADA chooses where to augment by clustering early-training losses once. It selects on loss magnitude, statically. We select on gradient agreement, repeatedly. The decoupled-VAE pipeline of Decision-Making with Auto-Encoding Variational Bayes fixes the generator before training several posteriors, which supports our choice of a frozen scorer. That work does not address how much to sample from the generator in the first place, which is our question.
All compute is accounted for honestly. The GPU-hours reported include generation, scoring, SNR probe backward passes, and retriever training.
Baselines
The naive reference point is the unadapted retriever, unsupervised E5 Base, evaluated zero-shot. It tells us how much any synthetic data buys at all.
The strongest existing method on this exact question is Promptodile, which generates unfiltered queries for the corpus with a small open model. It reports average BEIR nDCG@10 of 45.1 with Llama-3.2-3B, 46.2 with Phi-3-medium (14B), and 48.1 after switching the base retriever to unsupervised E5 Base (Study on LLMs for Promptagator-Style Dense Retriever Training). For reference, original Promptagator reports 44.7 unfiltered and 46.0 with round-trip filtering in that study. We rerun Promptodile with the identical generator, prompts, retriever initialization, and training-step count, then trace its accuracy as a function of the number of generated queries.
Beating it has two parts. First, we must match its full-budget score within noise while using at most one tenth of its generation compute. Second, at every matched budget we must score higher on the full-corpus average.
The strongest budget-aware competitor is DUQGen's static clustering plus Maximal Marginal Relevance selection, a diversity rule that penalizes picking items similar to ones already chosen. It reports beating zero-shot and unsupervised baselines on 16 of 18 BEIR datasets with an average relative improvement of 4% using 1,000 pairs. No absolute average nDCG@10 for DUQGen was found on Lacuna. We reimplement it with our generator and retriever at the same query counts as our controller, so the comparison isolates dynamic SNR allocation against static diversity.
We also report random subsampling of documents at matched budgets. RD³ found random real samples with teacher labels beat elaborate distillation methods on ImageNet-1K, and any allocation scheme must clear that bar.
The key ablation replaces $\mathrm{SNR}_c$ with mean retriever loss per cluster. This is the "obvious" closed loop. It also swaps the frozen scorer for round-trip filtering through the current retriever. A second ablation keeps SNR allocation but drops the frozen scorer, which separates the contribution of the allocation rule from that of decoupled validity. ElicitR (ElicitR) is reported as a zero-generation reference on BRIGHT, where it reaches 23.1 nDCG@10 at 3B parameters. It is not a like-for-like competitor on budget curves.
Why not just…?
Why not just generate for everything with a small model and skip the controller? This is the most serious threat. Promptodile shows a 3B generator without filtering already beats 137B Promptagator unfiltered, and 3B generation is cheap. If full unfiltered generation is affordable and our controller only matches it, the contribution shrinks to cost savings on large corpora. The budget curve settles this. If Promptodile's curve reaches its plateau at about the same query count our controller needs, or earlier, the adaptive idea fails and we will say so. The controller should matter most on large corpora and on domains where small generators produce many invalid queries. The latter is the regime where Balancing Cost and Effectiveness of Synthetic Data Generation Strategies for LLMs reports weaker augmenters causing substantial drops. That is why we break results out by domain, not only by average.
Why not just select once with DUQGen-style diversity? Static diversity is cheap and strong, and DUQGen's 1,000x data reduction shows most of the gain can come from choosing documents well. Its limitation is that the selection never learns what the retriever has already mastered. Its own ablation shows sensitivity to the number of clusters. If matched-budget DUQGen equals our controller, then the dynamic SNR signal adds nothing and the proposal is refuted on its central claim.
Why not just generate more where the retriever's loss is high? This is standard hard-example mining. We predict it underperforms even static diversity. Loss-driven allocation couples the generator's workload to the retriever's errors. Hallucinated or unanswerable queries produce persistently high loss that no update can fix, which is the synthetic-data version of the degenerate joint optima described by Lucas et al. The loss-steered ablation tests this directly. We also log the frozen scorer's validity rate on queries each controller chooses to generate. If the loss-steered loop does not concentrate budget on low-validity queries, our explanation is wrong even if the accuracy gap appears.
Why not just skip generation and use a generative regularizer? ElicitR avoids synthetic data entirely. It reports losing ground on symbolic domains such as math and code when its regularizer is trained on general text. Our method targets exactly the domain-grounding that generation provides. Comparing on BRIGHT's code and math subsets tells us whether any generation is needed there.
Proposed Experiments
The main experiment uses the 18 BEIR datasets. For each method, we plot nDCG@10 against total GPU-hours and against query count, from 1,000 queries up to Promptodile's full-corpus scale, with three seeds per point. Secondary evaluations use LoTTE with Success@5, the setting of UDAPDR's 64.5 to 74.8 improvement on Lifestyle, and BRIGHT with nDCG@10.
We also check whether SNR is a valid allocation signal on its own terms. At fixed intervals we compare $\mathrm{SNR}_c$ with the realized validation gain from a fresh batch of queries in that cluster. This validation uses a small number of real queries per BEIR dataset, reserved only for this analysis. A useful signal must predict marginal gain better than mean loss does, measured by rank correlation across clusters.
The decisive checks are three. First, our controller reaches Promptodile's full-budget BEIR average within 0.5 nDCG@10 at no more than 10% of its generation compute. Second, it beats matched-budget DUQGen on average. Third, the loss-steered coupled variant lands below static diversity, and it spends a measurably larger share of its budget on queries the frozen scorer rates invalid. If the first check fails, the method is not worth its complexity. If the second fails, dynamic allocation adds nothing beyond selecting once. If the third fails, the decoupling argument does not explain what we observe, whatever the accuracy numbers.
References
- Promptagator: Few-shot Dense Retrieval From 8 Examples
- UDAPDR: Unsupervised Domain Adaptation via LLM Prompting and Distillation of Rerankers
- DUQGen: Effective Unsupervised Domain Adaptation of Neural Rankers by Diversifying Synthetic Query Generation
- Tighter Variational Bounds are Not Necessarily Better
- Characterizing and Avoiding Problematic Global Optima of Variational Autoencoders
- Study on LLMs for Promptagator-Style Dense Retriever Training
- Variational Inference and Model Selection with Generalized Evidence Bounds
- Do We Need All the Synthetic Data? Targeted Image Augmentation via Diffusion Models
- Decision-Making with Auto-Encoding Variational Bayes
- Rectified Decoupled Dataset Distillation: A Closer Look for Fair and Comprehensive Evaluation
- ElicitR: Unlocking Latent Reasoning in Dense Retrievers via Generative Regularization
- Balancing Cost and Effectiveness of Synthetic Data Generation Strategies for LLMs