Scoring Synthetic Queries by Their Bayes Factor

Systems that train a small search model on queries invented by a large language model usually spend a second round of compute deciding which invented queries to keep. Promptagator trains a whole interim retriever just to throw away queries that fail to find their own source document. Other pipelines ask the language model to grade its own outputs, or run a large classifier over every candidate. This note proposes a much cheaper check built on an old idea from probabilistic similarity: a pair is similar when the second item becomes much more likely once you know the first, compared with how likely it is in general. For a synthetic query $q$ written for document $d$, the generator already computes how probable $q$ is given $d$ while it writes it. One extra short forward pass, with the document removed, gives how probable $q$ is with no document at all. The ratio of the two, a Bayes factor, rewards queries that are specific to their document. It penalizes generic queries ("what are the benefits of this?") and verbose ones, which the generator finds likely almost regardless of the document.

The claim is that this ratio is a better filter and training weight for synthetic retrieval data than either no filtering or the raw generation probability. It should also match or beat round-trip filtering at a small fraction of the cost, and it should avoid round-trip filtering's known failure of over-pruning small technical corpora. The idea is a two-sentence one. Use the generator's own likelihood of the query, divided by the query's likelihood without the document, to decide which synthetic pairs are worth training on. This costs a few percent of generation compute rather than a second training run.

Core Claim

Define, for a generator language model $G$ with few-shot prompt $P$, the score

$$ B(q,d) = \frac{1}{|q|}\Big[\log p_G(q \mid P, d) - \log p_G(q \mid P, \varnothing)\Big], $$

where $|q|$ is the number of query tokens and $\varnothing$ is the same prompt with the target document replaced by an empty placeholder. The first term is available for free from sampling. The second needs one forward pass over roughly $|P| + |q|$ tokens with no document, which is cheap: the expensive part of generation is reading a long document, and with prompt caching the shared prefix $P$ is computed once. This is the contextual Bayes factor $p(x_t \mid Q)/p(x_t)$ that Max-margin learning with the Bayes factor uses to score a target against a context set. That paper computes the factor with a variational autoencoder and Monte Carlo sampling on images. Here we observe that an autoregressive generator gives both terms exactly, in text, at almost no cost. The same reasoning underlies the odds-ratio "generative similarity" of GenSim, which needed an explicit probabilistic program to compute it. In our setting the language model plays the role of the program.

The concrete hypothesis has three parts. Filtering or weighting by $B$ beats the raw score $\log p_G(q\mid P,d)$, because the denominator cancels the generator's preference for frequent, length-matched phrasing. That preference is the failure Promptodile reports as sensitivity to query verbosity. Second, $B$ does not over-prune small specialized corpora. It is judged by the generator, which already knows the domain vocabulary. Round-trip filtering, by contrast, is judged by a freshly trained retriever that does not, and Promptagator documents that this filter degrades NFCorpus. Third, the gain holds when the pool of documents is already chosen for diversity, so the filter adds something beyond corpus selection.

Relation to Prior Work

The motivating evidence comes from the filtering literature, which disagrees with itself. Promptagator generates queries with a 137B model and filters by round-trip retrieval. Promptodile reports that this filter raised the BEIR average from 44.7 to 46.0 nDCG@10. It also finds that unfiltered queries from Phi-3-medium (14B) reach 46.2, which suggests filtering may be unnecessary with modern generators. UDAPDR avoids filtering by training several cross-encoder rerankers on up to a million noisy queries and distilling them. This moves the overhead into training rather than removing it.

Two lines of work spend even more compute on verification. GENERATE-PAIRWISE asks an LLM judge to re-classify each pair. DISCO runs a DeBERTa teacher over every candidate counterfactual. Our proposal differs from all of these in where the check comes from: it reuses quantities the generator already computes, adds no trained model, and adds no judging prompt.

One finding constrains the design. Promptodile shows that removing filtering is a strong default. So the filter must earn its place against "do nothing," and it must not merely remove data.

Two pieces of work supply the method. Probabilistic vs. Geometric Similarity Measures for Image Retrieval argues that a fixed distance threshold ignores how densely populated each region of the space is, and that a likelihood ratio adapts to it. The denominator $p(q\mid\varnothing)$ plays exactly this density role for queries. The pair-likelihood-ratio models of Bayesian Modeling of Facial Similarity and LSSL fit Gaussians to embedding differences, which gives an embedding-space alternative that we test as a competing scorer. DUQGen addresses overhead from the other end, by choosing 1,000 diverse documents with clustering and maximal marginal relevance. We treat it as complementary: it chooses which documents to generate for, and $B$ chooses which generated queries to keep.

Research Program

We first over-generate a pool of synthetic queries. For each BEIR corpus, we sample several queries per document from two open generators used by Promptodile, Llama-3.2-3B and Phi-3-medium, using the same 8-shot prompts. Every sample is logged with its conditional log-probability, and we compute the no-document log-probability with a cached prompt prefix. Compute is measured in GPU-seconds for every stage, so each filter's cost is reported next to its accuracy.

We then compare scorers on this one fixed pool. The scorers are: our Bayes factor $B$; the raw length-normalized conditional log-probability; cosine similarity between query and document under an off-the-shelf encoder (E5 base); an LSSL-style Gaussian log-likelihood ratio over encoder difference and sum vectors, fitted on the highest-$B$ pairs; and Promptagator-style round-trip filtering. We also test a harder denominator, which averages $p(q\mid d')$ over a few documents $d'$ from the same cluster as $d$. This costs several document-length passes and tests whether specificity against near neighbors matters more than specificity against nothing.

Each scorer is used two ways. Used as a hard filter, it keeps a fixed fraction of the pool (100, 75, 50 and 25 percent). Used as a soft weight, it scales each pair's contribution to the contrastive training loss, as Gap 5 of the file suggests.

Before training any student, we check whether the scores mean anything. Using BEIR's human relevance judgments, we compute how well each scorer separates true query–document pairs from the same queries paired with BM25-retrieved non-relevant documents, reported as area under the ROC curve. This tells us whether $B$ measures relevance or just fluency, independent of any student-training noise.

Baselines

The naive reference point is the Promptodile recipe with no filtering. We train the same dual-encoder student, initialized from the same checkpoint, on all generated queries. Promptodile reports 45.1 average nDCG@10 for Llama-3.2-3B and 46.2 for Phi-3-medium, both unfiltered. It also reports 48.1 when the student is initialized from unsupervised E5 base. Promptodile's exact dataset subset and averaging convention will be reproduced rather than assumed. We rerun this baseline ourselves on identical data because it is the strongest competitor: it is the cheapest possible pipeline, and it already matches filtered Promptagator.

The strongest existing filtering method is round-trip consistency from Promptagator. As reported in Promptodile, it moved the original Promptagator BEIR average from 44.7 to 46.0 nDCG@10. We implement it on the same pool, with the same interim-retriever architecture and the same top-$k$ choices as the original. Its interim-training compute is charged to it.

The ablation that isolates our key ingredient is the raw conditional log-probability filter: the same numerator with no denominator. Any gain of $B$ over this filter is attributable to dividing by the query's marginal likelihood. A second ablation compares the empty-document denominator with the same-cluster denominator.

The cosine-similarity filter, the LSSL-style Gaussian scorer and DUQGen-style document selection are run on the same pool, with the same retention fractions and the same student training budget. All methods share seeds; we use three seeds per configuration and report the spread.

No labeled development data is used to tune any threshold. Every method is reported at the same fixed retention fractions, and a single operating point (keep 75 percent) is declared in advance. Beating the baselines means a higher average nDCG@10 than the same-generator unfiltered student and the raw-log-probability filter, by more than the seed spread, at the pre-declared operating point. It also means matching or exceeding round-trip filtering's average while using under a tenth of its added compute, with no per-dataset loss on NFCorpus relative to no filtering.

Why not just…?

Why not just skip filtering, as Promptodile does? This is the most serious threat. Promptodile's 46.2 with no filter shows the average benefit of any filter may be small. Our answer is not to assume a benefit but to measure it: at matched retention, with seeds, and per dataset, where Promptodile reports verbosity sensitivity that an average can hide. If $B$ does not beat unfiltered training beyond seed variance, we will report that modern small generators make query-level filtering unnecessary. That would be a clean, compute-backed answer to the filtering contradiction in Gap 2. The soft-weighting arm also keeps all data, so it cannot lose by discarding useful examples.

Why not just filter by cosine similarity under a good off-the-shelf encoder? This costs one encoder pass per pair and needs no generator access, so it is a real alternative. Its weakness follows from the geometric critique in Probabilistic vs. Geometric Similarity Measures for Image Retrieval. A fixed cosine threshold favors queries that copy the document's surface words, which is the same lexical bias that BEIR warns is baked into many relevance labels. Such a filter may therefore reject abstract but valid queries. The pair-separation analysis and the per-dataset results on argument and scientific corpora (ArguAna, Touché, SciFact) will show whether this happens. If cosine filtering matches $B$, it is the simpler choice, and we will say so.

Why not just use the generation log-probability itself? It is free, and it is the obvious probabilistic filter. However, it scores how natural a query is as much as how specific it is to its document. Generic and length-matched queries get high scores for reasons unrelated to the document, which is exactly what the denominator removes. The numerator-only ablation tests this directly. If the two perform the same, the Bayes-factor framing adds nothing in practice.

Why not just distill a reranker ensemble, as UDAPDR does? UDAPDR improves LoTTE Lifestyle Success@5 from 64.5 to 74.8. But it does so by generating up to a million queries and training several DeBERTa-Large rerankers. That is an overhead budget our setting is designed to avoid, so we do not run it at full scale. We do run a single-reranker version at a matched compute budget, as a cost-normalized reference point.

Proposed Experiments and Decisive Checks

The main experiment trains students on the BEIR datasets used by Promptagator, under every scorer, retention fraction and generator described above. We report average and per-dataset nDCG@10 together with total GPU-seconds per pipeline.

Three checks decide the claim. The first is NFCorpus: round-trip filtering is documented to hurt there, and $B$ should not. The second is the pair-separation analysis on human relevance judgments, where $B$ should separate relevant from non-relevant pairs better than the raw log-probability; if it does not, the idea fails before any student is trained. The third is a DUQGen-style low-budget setting. We over-generate several queries for each of 1,000 diverse documents and keep the single highest-$B$ query per document. This tests whether the Bayes factor helps most when data is scarce, which is where DUQGen shows that every pair matters. DUQGen reports wins on 16 of 18 BEIR datasets with 1,000 pairs.

A secondary transfer test applies the same score to counterfactual generation, in the style of DISCO. There, $B$ would compare the likelihood of a revised hypothesis under the target label with its likelihood without the label. We would compare this against DISCO's DeBERTa-teacher filtering, whose counterfactuals flipped the label 83 percent of the time.