Spend Synthetic Queries Where the Relevance Function Bends

Teams that train small retrievers or classifiers on data written by a large language model usually spread the generation budget evenly: every seed document gets the same number of synthetic queries, and a filter afterwards throws away the bad ones. Nonparametric statistics settled long ago that even allocation is wasteful when the function being learned is flat in some places and sharp in others. Spatially adaptive smoothing is the family of methods that sample densely and use narrow windows where the function changes quickly, and sample sparsely with wide windows where it is flat. This note proposes to use that principle to decide, before generating anything, how many synthetic queries each region of a document collection should receive.

The central difficulty is telling apart two reasons a region can look hard. A region can be genuinely complex, meaning the student's relevance scores change quickly between neighboring documents. Or it can be noisy, meaning the teacher writes unreliable queries there. Loss-based active generation cannot tell these apart, because both produce high loss. We therefore split a cheap pilot signal into two parts. The first is variation across neighboring documents, which we treat as complexity and spend on. The second is variation across repeated queries for the same document, which we treat as noise and do not spend on. The claim is that this allocation reaches the retrieval quality of uniform generation with a fraction of the teacher calls, and that the complexity/noise split is the ingredient that makes it work.

Core Claim

Consider a synthetic-query pipeline for dense retrieval, the setting of Promptagator. The budget is a fixed number of teacher generations, and every pilot call counts against it. We claim that an allocation rule has three properties. It is proportional to document density times a locally estimated complexity. It is estimated in a cheap document embedding space. It discounts regions whose pilot signal is dominated by variation within a single document. Under these conditions, the rule matches the average BEIR NDCG@10 of uniform open-model generation at roughly half the generation budget. It should also beat loss-ranked and diversity-stratified selection at equal budget.

This differs from the closest work in three ways.

Local Function Complexity for Active Learning via Mixture of Gaussian Processes supplies the method. It learns a locally optimal length scale by letting Gaussian-process experts of different bandwidths compete under a gating network. It then turns that length scale into a non-uniform optimal sample density, with reported sample savings of 42% on the Doppler function and 31% on a 36-dimensional force field. That work assumes constant noise, draws from a fixed candidate pool or physics simulator, and never touches generative models. We make noise vary by region, since noise level is exactly what distinguishes a bad-teacher region. We also replace pool acquisition with generator budget allocation.

Towards Active Synthetic Data Generation for Finetuning Language Models motivates the proposal and is its main competitor. It steers the teacher toward seeds on which the student has high loss and reports 1.3x to 2x data efficiency over random selection. It uses no notion of neighborhood, density, or smoothness. Its criterion is a single loss value per example.

DISCO selects ambiguous seeds with Data Cartography. It still over-generates candidates and then filters them with a teacher model, so the rejected generations are pure overhead.

Research Program

The pipeline runs in three stages, and the pilot stage is charged to the budget.

The first stage embeds all documents with an off-the-shelf encoder such as Contriever or E5, then projects them to a low-dimensional representation. The projection matters because local smoothing breaks down in 768 dimensions. Nonparametric Classification on Low Dimensional Manifolds using Overparameterized ConvResNeXts gives a theoretical reason to expect the relevant difficulty to scale with the dimension of the underlying manifold, not the dimension of the embedding.

The second stage is a pilot round spending about 10% of the budget. We draw a random subset of documents and ask the teacher for two queries per document. We train a pilot student and record, for each query, the rank of its source document. This rank is the scalar signal $r(x)$ over document position $x$.

For each region we then estimate two quantities. The first is within-document variance, meaning the disagreement between a document's two queries, which serves as a local noise estimate $\sigma^2(x)$. The second is between-neighbor complexity $C(x)$. To estimate it, we fit the mixture-of-bandwidths gate from the LFC paper to $r(x)$ after subtracting the within-document component. A small winning length scale means high complexity.

The third stage spends the remaining budget with per-document query counts drawn from

$$q(x) \propto p(x)\, C(x)^{\gamma}\, \big(1+\sigma^2(x)/\bar\sigma^2\big)^{-\beta},$$

In this rule, $p(x)$ is the estimated document density, and $\gamma \ge 0$ is the complexity exponent. Setting $\gamma=0$ gives density-proportional, effectively uniform, allocation. The factor $\beta \ge 0$ down-weights regions where the teacher is unreliable, relative to the corpus-average noise $\bar\sigma^2$. Both exponents are tuned on held-out BEIR datasets and then frozen.

The product form follows the classical local bias-variance argument that underlies the LFC density. Where we depart from that argument is the noise term: the classical result would send more samples to noisy regions. Here the noise is generator noise, and more generations there return mostly unusable data.

The analysis component asks whether $C(x)$ is stable across pilot seeds and embedding choices. It also asks whether, on a one-dimensional Doppler problem, the estimated allocation recovers the density the LFC paper reports as optimal. This sanity check validates the estimator before any language model is involved.

Baselines

The naive reference point is uniform generation with a small open teacher, exactly as in Study on LLMs for Promptagator-Style Dense Retriever Training. That paper reports an average NDCG@10 of 45.1 with Llama-3.2-3B and no round-trip filtering, 46.2 with Phi-3-medium, and 46.0 for the original 137B Promptagator with filtering. We rerun Llama-3.2-3B uniform generation on the same document subsets and retriever initialization. We treat our reproduced number as the reference, reporting 45.1 alongside it.

We run a random-subset baseline at each reduced budget. This separates "fewer queries" from "better-placed queries."

The strongest competitor is loss-ranked seed selection in the style of Kessler et al. Towards Active Synthetic Data Generation, adapted to retrieval by ranking documents by pilot-student rank loss. It uses the same pilot round, the same total budget, and the same tuning budget for its one free parameter, the selection temperature. We are not aware of a reported retrieval number for this method, since that paper evaluates only reasoning benchmarks. Beating it means higher average NDCG@10 at equal budget on at least the 50% and 25% budget points, with the gain holding across three seeds.

We also include a diversity baseline, which applies k-means clustering to the same embeddings and allocates equally per cluster. A loss-times-diversity hybrid combines the two. We include a post-hoc filtering baseline: generate uniformly at full budget, apply round-trip filtering, and account for every discarded generation as spent.

Two ablations isolate the key ingredient. The first sets $\beta=0$, keeping complexity without the noise discount. The second sets $\gamma=0$, keeping density and the noise discount without complexity.

BM25 serves as the zero-cost floor. BEIR reports it as a strong zero-shot baseline, but no reported average number for the Promptodile subset was found on Lacuna, so we measure it ourselves.

Why not just…?

Why not just rank seeds by student loss? This is the real threat, since the Kessler et al. result shows loss ranking already buys up to 2x efficiency. Our argument is that high loss mixes two causes. Documents the teacher cannot write good queries for have high loss, and so do documents the student genuinely confuses with their neighbors. Only the second kind rewards extra generation. The $\beta=0$ ablation against the full method tests this directly. If loss ranking matches the full method, and the noise discount adds nothing, the complexity machinery is not earning its cost and we would report that.

Why not just shrink the teacher? Promptodile shows a 3B model can replace a 137B one. That lowers the cost of each call but still spends the calls uniformly. The two approaches stack, so our experiments run on top of the 3B teacher, and every gain is measured beyond that already-cheap setting.

Why not just cluster the corpus and stratify? Clustering fixes redundancy but treats a flat cluster and a sharp cluster alike. The $\gamma=0$ ablation and the k-means baseline show whether complexity matters beyond coverage. If stratification alone closes the gap, the finding is that coverage explains the gains, which is still useful.

Why not just generate in bulk and filter afterwards? DISCO shows filtering improves quality, but the discarded generations are already paid for. Promptagator's filtering starved small corpora such as NFCorpus. Under our accounting, filtering pays full generation cost, and NFCorpus is reported separately to test the data-starvation case.

Proposed Experiments

The main experiment traces NDCG@10 against teacher generations at 25%, 50%, and 100% of the Promptodile budget. It covers the BEIR datasets used in the Promptodile study, with three pilot seeds per point. The reported results are per-dataset scores, the average, and the area under the budget curve.

The decisive checks are three. First, the full method must reach the reproduced uniform 3B average at 50% budget. Second, it must beat loss ranking at equal budget. Third, removing the noise discount must lose most of that margin. If the first check fails, the idea fails. If only the third fails, the contribution shrinks to density-aware allocation.

Secondary checks cover transfer and stability. We rerun the allocation with Phi-3-medium as teacher, against its reported 46.2. We measure how well $C(x)$ agrees across two embedding models. We run the one-dimensional Doppler recovery test described above.

A single transfer test to counterfactual generation on SNLI, following DISCO's setup and stress-test metrics, asks whether the same allocation reduces the number of rejected candidates per accepted counterfactual.