Priming Retrievers to Adapt from Hundreds of Synthetic Queries

Teaching a search model a new domain, such as financial forums, biomedical abstracts, or argument retrieval, currently depends on volume. A large language model writes queries for up to 100,000 documents in the target collection. A standard retriever is then fine-tuned on those query–document pairs, often after extra filtering passes that cost more model calls. The retriever here is a dual encoder, meaning two networks that map queries and documents into a shared vector space and rank documents by similarity. Most of the cost is paid again for every new domain, and the reason is that the student starts from weights trained for average performance rather than for adapting quickly.

We propose to meta-learn the retriever's starting point. Meta-learning here means optimizing the initial weights so that a short, fixed adaptation procedure works well across many training domains. The adaptation procedure we optimize for is exactly the one the deployed system will use: a few hundred generated queries, a small set of trainable low-rank adapter weights, and hard negatives. Hard negatives are documents that look relevant but are not. We find them for free by searching the target corpus with the current model and by mixing their embeddings, so no language model is asked to write them. The bet is that such a "primed" retriever reaches the quality of a 100,000-query pipeline with roughly one to two orders of magnitude fewer generated queries. A second bet is that the priming makes it stable under training on hard negatives alone, which current pipelines cannot do.

Core Claim

The claim is that the volume of synthetic data a domain needs is partly a property of the student's initialization, and that this part can be bought down once, ahead of time. The property can be measured. For each new domain, we plot retrieval quality against the number of generated queries. A primed initialization should shift that curve left by at least 10×, and the shift should not appear for a multi-domain model trained on the same data without the nested optimization.

Three lines of prior work motivate this. The first shows that synthetic queries can replace human labels but uses volume to do it. Promptagator prompts a 137B model with 8 examples to generate queries over whole target corpora and adds a round-trip filter, a temporary retriever that discards queries it cannot map back to their source document. Promptodile shows that 3B–14B open models, with no filtering, match it. UDAPDR generates up to a million queries and distills several rerankers into a retriever. In all three, the student is an ordinary pretrained checkpoint. We keep their generator choice, using a small open model following Promptodile, and attack the volume instead.

The second line shows that the few-shot regime is where standard retrievers break. Rethinking Dense Retrieval's Few-Shot Ability reports a 23.1% average drop on novel classes at 5 shots relative to full-shot training, with variance more than double that of 100 shots. It tests only conventional fine-tuning.

The third line supplies the method and also limits its design. Know Where You're Going meta-primes a backbone for adapter tuning and gains up to 4.96 F1 on Hindi named-entity recognition. Its key negative result is that ordinary MAML, which updates all weights in the inner loop, did not help downstream adapter tuning. We take that result as a design rule: the inner loop must reproduce the deployment-time adaptation exactly. That paper used a classification loss with no negatives. Our inner loop is contrastive and includes the hard-negative scheme the deployed system will use.

Research Program

The meta-objective is the standard two-level one. Here $\theta$ is the backbone weights and $\phi_0$ is the initial adapter weights. $U_K$ runs $K$ adapter-only gradient steps of a contrastive loss on a support set $S_d$ of generated queries for domain $d$, using self-mined hard negatives:

$$\min_{\theta,\phi_0}\ \mathbb{E}_{d}\Big[\mathcal{L}_{\text{rank}}\big(\theta,\ U_K(\theta,\phi_0;S_d);\ Q_d\big)\Big]$$

$Q_d$ is a held-out query set for the same domain, and $\mathcal{L}_{\text{rank}}$ is the contrastive ranking loss evaluated after adaptation. To keep cost manageable, we use a first-order approximation of the meta-gradient. Because only adapter weights change in the inner loop, the memory cost of unrolling stays small.

Inside $U_K$, hard negatives come from two cheap sources. The first is passages from the target corpus that rank near the top for the generated query under the current encoder, excluding the source passage and the next few nearest neighbors. That exclusion guards against unlabeled relevant passages being treated as negatives, the false-negative problem. The second source is embedding-level mixtures of those passages with each other and with the query. This follows MoCHi, which creates harder negatives on the fly by interpolating feature vectors in vision contrastive learning, and SSCL, which adds sampling weighted toward closer negatives plus a debiasing term against false negatives. Both were developed for images. We use them only to make the inner loop's negatives harder, not as a substitute for semantic queries.

Meta-training domains come from two sources. One is the 30 base relation classes of FewDR. The other is a set of BEIR corpora that are held out from evaluation, with queries for them generated once by Llama-3.2-3B. Real labeled queries serve as $Q_d$ only when the domain is a meta-training domain. Evaluation domains never contribute labels to meta-training.

Cost accounting is part of the program, not an afterthought. For every method, we will report the tokens generated, the GPU-hours spent on generation, filtering, and adaptation per domain, and the one-time meta-training cost. From these we compute the break-even number of new domains after which priming is cheaper overall.

Baselines

The naive reference point is the base dual encoder applied zero-shot. We will use E5-base (unsupervised), because Promptodile identifies it as the strongest starting retriever in its study. A second naive point is the same checkpoint adapted with the same small query budget and plain in-batch negatives. In-batch negatives means the other documents in the training batch serve as negatives. This second baseline shows what a small budget buys without any of our ingredients.

The strongest existing method is Promptodile's pipeline: generation over up to 100,000 documents with an open 3B–14B model, no filter, and full fine-tuning of E5-base. It reports 48.1 average NDCG@10 on BEIR (normalized discounted cumulative gain over the top 10 results, the standard ranking metric) (Promptodile). For reference, Promptagator's 8-shot retriever reports 47.8 nDCG@10. We will rerun Promptodile's pipeline ourselves on the same BEIR subsets, with the same generator, the same backbone, and a matched hyperparameter search. The comparison therefore does not rest on numbers copied across papers.

Beating it means two things. First, matching its NDCG@10 within one point using at most 1,000 generated queries per domain. Second, cutting total per-domain generation-plus-adaptation compute by at least 10×. A smaller compute cut that still saves money after a stated number of domains would count as partial success.

The ablation that isolates our key ingredient is a multi-domain baseline. It trains the same backbone and adapters on the same pooled meta-training data, with the same hard-negative scheme and the same total compute, but without the nested optimization. Both are then adapted identically on new domains. A second ablation removes the hard negatives from the inner loop while keeping them at deployment, which tests whether matching the negative regime in the inner loop matters. A third runs full-weight MAML, which tests the protocol-matching rule in the contrastive setting.

For the stability question, the reference is SyNeg's hybrid recipe. SyNeg has GPT-4o write hard negatives, reports +2.9 and +3.0 NDCG@10 on COCO-DR and BGE-large, and needs dilution with BM25 negatives (about 0.7 synthetic negatives per positive) to avoid collapse. We run it on its five reported datasets with the same student, and compare against our primed model trained on hard negatives alone.

On LoTTE, the reference is UDAPDR's reported Success@5 of 74.8 on Lifestyle, against 64.5 for zero-shot ColBERTv2. Success@5 is the fraction of queries with a relevant document in the top five. Since UDAPDR's student is ColBERTv2, a multi-vector retriever rather than a single-vector dual encoder, this comparison indicates the cost and accuracy trade-off rather than a controlled head-to-head.

Why not just…?

Why not just generate fewer queries with a small open model and fine-tune normally? This is the cheapest competitor. If the learning curve of a standard checkpoint is already flat by about 1,000 queries, priming has nothing to buy. The file does not report a sub-10,000-query curve for Promptodile's setup. The FewDR results, the 23.1% few-shot drop and doubled variance, suggest the curve is steep at small budgets, but they come from relation classes rather than BEIR corpora. Our first experiment measures exactly this curve. If the plain checkpoint reaches within one point of its 100,000-query result at 1,000 queries, we will drop the volume claim and report the stability result alone.

Why not just pretrain on many synthetic domains without the nested optimization? This is the real threat. Broad multi-domain training could produce an initialization that is just as easy to adapt, and it would be simpler. The compute-matched multi-domain ablation settles the question. If it closes most of the gap, the conclusion is that diversity of meta-training data, not meta-learning, reduces synthetic volume. That is still a useful finding about overhead, and we would report it as such. Meta-Learning Requires Meta-Augmentation warns that meta-learners can memorize their training tasks, so we also require gains on FewDR's novel classes, not only on held-out BEIR corpora.

Why not just add latent hard-negative mixing to ordinary fine-tuning? MoCHi and SSCL already make negatives harder at essentially no generation cost. However, their gains were shown for image representations: MoCHi improves ImageNet-100 top-1 accuracy from 78.0 to 79.0, and SSCL improves SimCLR on CIFAR-100 from 52.02 to 60.01. Interpolated embeddings cannot supply the query intent that a domain's generated queries carry. We run mixing-plus-standard-fine-tuning at every query budget. Our claim survives only if priming adds to mixing, rather than mixing alone explaining the gain.

Why not just tune adapters on a standard checkpoint? Know Where You're Going reports a 4.96 F1 deficit for unprimed adapters relative to primed ones in Hindi. That evidence comes from classification, though, and we test it directly: the plain-adapter baseline is the second naive point described above.

Proposed Experiments

The first decisive check is the budget curve. On the Promptodile BEIR subsets, we plot NDCG@10 against generated queries per domain at 64, 256, 1,000, 10,000, and 100,000, for four models: the plain checkpoint, the multi-domain ablation, mixing plus fine-tuning, and the primed model. Three seeds are run per point, because FewDR reports high variance at small budgets. The primed model must lie above all others at 1,000 queries or fewer and come within one point of the 100,000-query Promptodile rerun.

The second check is FewDR novel-class recall@10 at 5 and 20 shots. The question is how much of the reported 23.1% drop priming recovers, compared with that paper's query-generation augmentation, which reports a 16.9% gain.

The third check is stability. On SyNeg's five datasets, both the standard and the primed student are trained on hard negatives only, with no BM25 dilution. We log gradient-norm variance and the divergence rate across seeds. Priming should remove the collapse SyNeg documents. If it does not, the stability claim is false, and we will say so.

The fourth check tests whether priming makes the round-trip filter unnecessary. It applies the filter on top of the primed model's budget runs, including NFCorpus, where Promptagator reports that filtering discarded valid queries and hurt results.

Finally, the cost table reports generation tokens, GPU-hours, and the break-even domain count, so that a reader can decide whether the one-time meta-training cost pays off.