Spending the Generation Budget Where the Target Domain Is Uncovered

Teams that adapt a search model to a specialized domain, such as nutrition science, scientific claims or technical forums, usually do it with synthetic data. A large language model reads documents from the target collection and writes plausible questions about them. A small retriever is then trained on these (question, document) pairs. Current pipelines pick the documents to prompt with at random and generate enormous volumes: up to a million queries per domain in UDAPDR, and millions from a 137-billion-parameter model in Promptagator. Afterwards they throw much of it away through consistency filters. Most of that compute goes to regions of the domain that are already well covered.

We propose to make generation adaptive. After each small round of generation, we measure how far the synthetic questions are from covering the target document collection, using optimal transport. Optimal transport is a way of comparing two point clouds by the cheapest way to move one onto the other. We then prompt the generator next on the documents whose coverage would most reduce that distance. The quantity that tells us where to go next is a standard by-product of the transport computation: its dual potential. This is a per-location score that, to first order, says how much the total transport cost would drop if one more synthetic point were placed there. It can be maintained cheaply with stochastic updates, so choosing where to generate costs almost nothing compared with generating. We stop when the distance to held-out target documents stops falling. The claim is that this matches the retrieval quality of brute-force pipelines with a small fraction of their generator calls, and that it degrades less gracefully under hard filtering in small domains.

Core Claim

Existing optimal-transport data selection works on fixed piles of data. TAROT greedily picks a subset of a static candidate pool that minimizes transport distance to a target set, using whitened gradient features, and stops when that distance stops decreasing. It shows that a well-chosen 24% of synthetic GTA5 beats training on all of it for Cityscapes segmentation. FDMat does the same for labeled image classification, using class centroids and Sinkhorn matching. Both motivate our proposal, because they show unaligned synthetic data can hurt. Neither reduces generation cost, however, since the pool must already exist.

Synthetic-data pipelines for domain retrieval have the opposite profile. They address the domain gap but spend generation blindly. We connect the two by moving the transport computation upstream of generation. The acquisition score is the first variation of an entropic transport divergence between the weighted synthetic set $S$ and the target corpus $T$, both embedded by the current student retriever. Concretely, with source potential $f$ extended to any location $x$ by the soft c-transform,

$$f(x) = -\varepsilon \log \sum_{y \in T} b_y \exp\big((g(y) - c(x,y))/\varepsilon\big),$$

where $g$ is the target-side potential, $b_y$ the target weights, $c$ the embedding cost and $\varepsilon$ the smoothing strength. Placing a small amount of new synthetic mass at $x$ changes the divergence by roughly $f(x)$ minus its self-term. We therefore rank unprompted corpus documents by this score, using the seed document's embedding as a proxy for where its generated query will land. Only the top of that ranking is sent to the generator.

The potentials are updated with the mini-batch dual stochastic gradient methods of Stochastic Optimization for Large-scale Optimal Transport and Stochastic Optimization for Regularized Wasserstein Estimators. Those papers show per-step cost independent of dataset size. This matters because target corpora have hundreds of thousands to millions of passages, and we recompute after every round.

A second change replaces discarding with weighting. Promptagator discards any query that fails to retrieve its own seed passage, which the file notes depletes small domains such as NFCorpus. DISCO over-generates and rejects through a teacher classifier. We instead give each synthetic pair a transport mass $w_i$ proportional to a cheap round-trip consistency score. A doubtful query therefore still counts partially toward coverage, and the region it sits in is not immediately re-targeted. The retriever itself is trained on the same weights.

Research Program

The system runs in rounds. It begins as in Promptagator, with 8 in-domain example pairs, and a round-zero batch of a few thousand queries from random seeds. Each round then does four things. It fine-tunes the student retriever briefly on the weighted synthetic set. It re-embeds the synthetic queries and a streaming subsample of the target corpus. It takes a few thousand stochastic dual steps, warm-started from the previous potentials. Finally, it scores a candidate pool of unprompted passages and sends the top-scoring batch, with a diversity temperature, to the generator. The same k-fold held-out transport-distance criterion that TAROT uses for stopping selection here decides when to stop generating.

We reuse TAROT's released code, vita-epfl/TAROT, for the whitening and distance computations. However, we use retriever embeddings rather than per-example gradients for the ground cost, to avoid the gradient overhead the file flags as prohibitive inside a generation loop.

Three analyses run alongside the main result. First, we measure proxy drift: how far each generated query's embedding lands from its seed passage's embedding, and whether large drift predicts wasted rounds. If drift is large, we will fit a small regressor from passage embedding to expected query embedding and score with that instead. Second, we compare scoring in the fixed initial embedding against the retriever's updated embedding, because a good retriever geometry is itself what we are learning. Third, we test sensitivity to the smoothing strength $\varepsilon$ and the step size, which the file lists as known weak points of stochastic dual methods.

A constraint shapes the choice of testbed. TheoremQA and the medical reading-comprehension study argue that expert-domain failures come from latent principles that surface embeddings do not capture. On TheoremQA, even supplying theorem definitions did not significantly help. Coverage in embedding space is therefore a poor proxy for "teaching a model to apply a theorem," and we do not claim it is. Retrieval is the setting where embedding geometry is the task itself, so it is the honest place to test coverage-driven generation first. TheoremQA's 800 questions provide no training distribution, so we leave reasoning domains to future work rather than overclaim.

Baselines

The naive reference point has two parts. The first is the zero-shot retriever with no synthetic data. The second is random-seed generation at matched budgets, meaning the same generator, prompts, retriever and training recipe, with seeds drawn uniformly. Random generation run to the full budget is the brute-force regime.

The strongest existing system is UDAPDR. It uses GPT-3 seed prompts, Flan-T5 XXL generating up to one million queries, several cross-encoder rerankers trained on subsets, and distillation into ColBERTv2. The paper reports that Success@5 on LoTTE Lifestyle rose from 64.5 to 74.8 at 35ms query latency. We will rerun its public recipe with Flan-T5 XXL as generator for all methods, report its quality at its own full budget, and then truncate its query set to our budgets. On BEIR, the reference is Promptagator, which reports 47.8 average nDCG@10 for its 8-shot dual encoder, 45.5 zero-shot, and 52.8 with reranking. The file has no reported per-dataset figure for NFCorpus or SciFact, so we will report our own reproduction there. Because we cannot run a 137B model, we will present Promptagator's recipe with the shared open generator and state that difference plainly.

The key competitor is generate-then-select. Here we generate a large random pool and apply TAROT-style transport selection at equal training-set size. This isolates whether steering saves generation or merely matches selection after the fact.

Three ablations isolate the ingredients. One replaces the transport score with cluster-stratified seeding. One keeps transport steering but uses hard round-trip rejection instead of soft weights. One keeps random seeds but uses only the transport stopping rule.

All methods get the same generator, the same retriever architecture, the same number of retriever training steps per retained example, and the same hyperparameter-search budget on a held-out dev split. Cost is counted in generator tokens and total GPU-hours, including our potential updates. Beating UDAPDR means one of two things. The first is matching its Success@5 within noise at no more than 10% of its generator tokens. The second is exceeding it at an equal budget on at least three of the five LoTTE domains, with confidence intervals across three generation seeds.

Why not just…?

Why not just cluster the corpus and sample seeds evenly from each cluster? This is the most serious threat. It is cheap and density-aware, and it may capture most of the benefit. Our method differs in two ways. It reacts to what the generator actually produced, including off-target drift and low-consistency outputs. It also updates as the retriever's geometry changes. Stratification fixes a quota in advance. The stratified ablation, run at every budget, settles the question directly. If it matches transport steering within noise, the honest conclusion is that simple static coverage suffices, and we will report that as the finding.

Why not just generate a large pool and then select with TAROT? Selection can improve quality, as TAROT's GTA5 result shows. But it pays the full generation bill first, and the overhead is what we target. The generate-then-select comparison at equal generator tokens shows whether selecting after the fact ever beats steering beforehand. If it does, the steering idea is wrong.

Why not just pick the passages the retriever is least confident about? AvAtar found that local uncertainty sampling ignores global alignment structure. Its global transport-utility gradients beat entropy and density sampling on all eight of its alignment benchmarks, by up to 2.8% MRR. That evidence comes from matching tasks, not generation, so we include an entropy-seeded variant as an extra baseline rather than assume the result transfers.

Why not just generate a million queries, since compute keeps getting cheaper? TAROT and DISCO show that more unaligned synthetic data can hurt: DISCO's 75,000 filtered examples beat pipelines with over a million unverified ones. Beyond Quantity documents early plateaus from quantity scaling. Our quality-versus-tokens curves test directly whether brute force is still rising at one million queries on LoTTE.

Proposed Experiments

We evaluate on the five LoTTE domains (Lifestyle, Technology, Writing, Recreation, Science) with Success@5. We also evaluate on BEIR SciFact, NFCorpus and Climate-FEVER with nDCG@10, chosen because they are specialized and, in NFCorpus's case, small enough for filtering to deplete. For each method we plot retrieval quality against generator tokens at budgets of 10k, 25k, 50k, 100k and the full run.

The decisive check has two parts. The first is whether transport-steered generation reaches UDAPDR's full-budget Success@5 within 25k to 100k queries while random seeding at the same budget does not. The second is whether steering beats cluster-stratified seeding at equal budget. Failing the first falsifies the efficiency claim. Failing the second means the dual potential adds nothing over simple coverage heuristics.

A secondary check targets small domains. On NFCorpus, soft weighting should beat hard rejection by a larger margin than on bigger corpora, because the depletion Promptagator describes should bite hardest there. Finally, the stopping rule is judged by how close its chosen budget lands to the best-performing point on the quality curve. It is useful only if it lands near the curve's knee without ever seeing labeled target queries.