A common way to build a cheap, deployable model today is to have a very large language model write training data for it, and then train a small "student" model on that data. In search, this works well. A 137-billion-parameter model writes queries for documents, and a 110-million-parameter retriever trained on those queries beats systems trained on half a million human labels. The bill, however, is paid upfront. That bill covers running the large writer over millions of documents, running extra checking passes to throw out bad examples, and only then training the small model. Papers report the accuracy of the student. They almost never report what the whole pipeline cost, or whether the same money would have been better spent on a slightly larger student, a smaller writer producing more data, or no checking at all. They also rarely check whether the chosen recipe still holds when the student is evaluated on a dataset nobody tuned on.
This note proposes a compute-matched study of that budget question for dense retrieval. Dense retrieval here means search by comparing learned vector embeddings of the query and the document. The central hypothesis is specific and falsifiable: filtering synthetic data is worth its cost only for small students, because noise tolerance grows with student capacity. The optimal split of a fixed budget therefore moves from "filter hard" toward "generate more, filter less" as the student grows. A second claim makes the study useful rather than descriptive. A budget allocation chosen on some benchmark corpora should transfer to held-out corpora. We test this by cross-validating the recipe itself, leaving one corpus out at a time, rather than only cross-validating the trained model.
Why this question, and why now
Three findings in the literature point in different directions, and that disagreement motivates the study. Promptagator has a prompted 137B model write queries for each target corpus. It keeps only queries for which a temporary retriever recovers the source document, a step it calls round-trip filtering, and it treats this filter as essential. The same paper notes that the filter is overly aggressive on small corpora such as NFCorpus. DISCO likewise argues that strict teacher filtering is what makes 75k counterfactuals beat much larger unfiltered sets. In contrast, Scaling Laws of Synthetic Data for Language Model finds that fine-tuning 1B–8B models on synthetic data is robust to moderate answer noise, so heavy verification buys little.
These papers trained students of very different sizes. Our reading is that they may not disagree at all. Small models may simply be less able to average out label noise. Nobody has tested that interaction directly, and it determines whether the expensive filtering stage should exist.
The capacity side is constrained by Scaling Laws of Synthetic Images for Model Training ... for Now. That paper trains vision models on images from text-to-image generators and finds that gains stop beyond about 4M synthetic images unless the backbone grows from ViT-B to ViT-L. If that ceiling generalizes, the sub-100M students that edge deployment demands may not absorb large synthetic corpora, especially out of domain. This is a negative result our design must be able to reproduce or refute.
UDAPDR builds the most elaborate version of the pipeline: GPT-3 seeds, Flan-T5 XXL generating up to a million queries, several large rerankers, and distillation into ColBERTv2. It cuts query latency from up to 20,600ms to 35ms. It reports no cost–accuracy curve, and it never tries a student lighter than ColBERTv2. TrueTeacher shows a similar pattern in factual-consistency checking. It distills a 540B labeler into a T5-11B student that beats the teacher on the TRUE benchmark (ROC-AUC 87.8 vs 84.9), but it never asks how small the student can go. All three supply the method we reuse. What they omit is the accounting.
Core Claim
We claim two things for dense retrievers trained on synthetic queries and evaluated zero-shot across heterogeneous corpora. First, the marginal value of a unit of compute spent on filtering decreases with student size, and at some student size it falls below the value of spending that unit on generating more unfiltered queries. Second, the compute-optimal allocation, once total cost includes serving cost amortized over the expected number of deployment queries, is stable enough across corpora to be chosen on held-out benchmarks without tuning on the target.
The accounting we use is
$$C_{\text{total}} = C_{\text{gen}}(G, N) + C_{\text{filter}}(N) + C_{\text{train}}(S, N_{\text{kept}}) + Q \cdot c_{\text{inf}}(S),$$
where $G$ is generator size, $N$ the number of generated queries, $N_{\text{kept}}$ the number surviving the filter, $S$ student size, $Q$ the expected number of deployment queries, and $c_{\text{inf}}$ per-query encoding and search cost. Generation and filtering costs are estimated with the standard approximation of about two floating-point operations per parameter per token processed, and checked against measured GPU-hours. The deployment volume $Q$ is the knob most papers silently set to zero. Varying it is what lets a larger student "pay for itself" or not.
Research Program
We first rebuild a Promptagator-style pipeline with open models. We cannot run FLAN 137B, so we use open instruction-tuned generators at roughly three scales, around 1B, 7B, and the largest we can afford near 70B, with Flan-T5 XXL as an additional point because UDAPDR uses it. The 8-shot prompt format follows Promptagator. Students are dual-encoder retrievers at roughly 10M, 30M, 110M, and 330M parameters, all initialized from comparable pretrained encoders. The filter is round-trip consistency with three strictness levels plus "off."
At each total budget level, we enumerate allocations over generator size, query count, filter level, and student size whose summed cost matches the budget. For each allocation we train a student per target corpus and record nDCG@10, the standard ranking metric for BEIR.
The analysis has three parts. The first fits the interaction between filter strictness and student size on held-out nDCG@10. This is the test of the central hypothesis. The second traces, for each student size, how held-out gains scale with the number of kept synthetic queries. This shows whether a capacity ceiling like the one Fan et al. report for images appears for small text encoders. The third is the recipe cross-validation. For each BEIR corpus in turn, we pick the best allocation using only the other corpora and report the regret on the held-out corpus. Regret is the nDCG@10 gap between the allocation we picked and the best allocation for that corpus in hindsight. This tests whether "best recipe" is a portable concept or a per-corpus accident.
Baselines
The naive reference point is BM25, plain lexical keyword matching, run zero-shot on every corpus with zero generation cost. BEIR shows that BM25 beats several dense retrievers, including DPR, ANCE, and TAS-B, out of domain. It also shows that docT5query, which expands documents with model-written queries before keyword indexing, beats BM25 on 11 of 18 datasets. BEIR as summarized in the research file gives no per-dataset averages for these systems. We therefore rerun both on identical corpora and report the docT5query expansion cost in the same accounting. Any synthetic dense allocation that does not beat docT5query at matched cost is a negative result for the whole approach, and we will report it as such.
The strongest existing method is Promptagator itself. The file reports an average of 47.8 nDCG@10 for the 8-shot version over 11 BEIR datasets, against 45.5 zero-shot. It reports 52.8 for Promptagator++ with a reranker, and a 1.2-point average advantage over ColBERTv2 and SPLADE v2. Our open-model replication of the 110M, strict-filter configuration is the internal anchor. All comparisons use the same 11 datasets, prompts, training steps, and hyperparameter search budget, which we fix at an equal number of trials per allocation. The reported 47.8 serves as an external sanity check, not as a compute-matched competitor, since its generator cost differs from ours. Beating the baseline means a cheaper allocation that equals our replication's held-out average, or an equal-cost allocation that exceeds it by more than the seed-to-seed spread.
On the five LoTTE domains, UDAPDR reports Lifestyle Success@5 rising from 64.5 to 74.8. We rerun a reduced UDAPDR, with a single reranker teacher and ColBERTv2 as the student, as the high-complexity point on the cost curve.
The key ablation removes filtering and spends its exact compute on additional generation, holding everything else fixed. A second ablation fixes total budget and swaps generator size for query count. A third uses an off-the-shelf retriever trained on MS MARCO human labels with no synthesis, which isolates whether synthesis matters at all at each student size.
Why not just…?
Why not just use BM25 or docT5query, which cost almost nothing to adapt? This is a real threat, particularly for the 10M–30M students, where a capacity ceiling could leave synthetic dense retrieval below keyword search. BEIR's evidence that BM25 beats unadapted dense models motivates exactly this concern. Promptagator's results suggest corpus-specific synthesis closes the gap at 110M, but not below that. Our cost curves include both lexical systems as points. If they dominate at small budgets or small $Q$, the honest conclusion is "use docT5query there," and we will report it.
Why not just always use the biggest generator? Fan et al. show that synthetic data can follow a much shallower scaling slope than real data (k ≈ 0.29 vs 0.5 for supervised ImageNet). This suggests the quality of each example and the number of examples trade off nontrivially, so it is not obvious that a 70B writer beats a 7B writer producing ten times as many queries. The matched-budget generator sweep answers this directly.
Why not just skip filtering and generate more, as SynthLLM suggests? We do not dismiss this. It is one arm of our central hypothesis, and it may win for 330M students. Promptagator's NFCorpus observation and DISCO's curated-beats-bulk result argue it will lose for small students. The filter-off ablation at each student size settles which regime holds where.
Why not just fit a scaling law on in-domain data and extrapolate? BEIR shows that in-domain strength on MS MARCO predicts out-of-domain behavior poorly. An allocation tuned in-domain may therefore not transfer. The leave-one-corpus-out regret measures exactly how much is lost by choosing a recipe without seeing the target.
Proposed Experiments and Decisive Checks
The benchmarks are the 11 BEIR datasets Promptagator used, including ArguAna, HotpotQA, Touché-2020, FEVER, Climate-FEVER, NFCorpus, and SciFact, scored by nDCG@10. The five LoTTE domains are scored by Success@5 and serve as a second held-out family that never informs allocation choices. We report three seeds per configuration. Costs are given in both estimated FLOPs and measured GPU-hours, and inference latency is measured on a single CPU core to reflect edge deployment.
Three outcomes decide the project. First, the central hypothesis is supported if the benefit of filtering over compute-matched extra generation is positive at 10M–30M and shrinks toward zero or turns negative at 330M, with the trend consistent across most BEIR corpora. It is refuted if the benefit is flat across sizes. Second, the capacity-ceiling question is answered by whether held-out nDCG@10 for the smallest students plateaus at a lower kept-query count than for 110M. Third, the recipe is portable if leave-one-corpus-out regret is smaller than the gap between the best and the median allocation. If regret is large, we report that compute-optimal synthesis must be tuned per target, which would itself be a useful negative result for anyone planning a synthetic-data budget.
References
- Promptagator: Few-shot Dense Retrieval From 8 Examples
- DISCO: Distilling Counterfactuals with Large Language Models
- Scaling Laws of Synthetic Data for Language Model
- Scaling Laws of Synthetic Images for Model Training ... for Now
- UDAPDR: Unsupervised Domain Adaptation via LLM Prompting and Distillation of Rerankers
- TrueTeacher: Learning Factual Consistency Evaluation with Large Language Models
- BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models