Neighborhood Batches as a Free Filter for Synthetic Retrieval Data

Small search models are now routinely trained on queries that a large language model writes for each document in a target collection. This works, but much of the cost does not go into writing the queries. It goes into checking them. Promptagator retrains a temporary retriever to test whether each query finds its source document. GENERATE-PAIRWISE asks the language model for a second, deliberately irrelevant query per document and then has the model judge both. Every one of these checks looks at a single document, or a single pair, in isolation. This note proposes the opposite. Group documents that are near neighbors into the same training batch. Then let the queries written for sibling documents do the checking work. A query that fits its neighbors' documents as well as its own is probably vague or hallucinated, so it should count for less. A sibling's query that is nearly a paraphrase of it is probably also relevant to its document, so that document should not be treated as a negative.

The claim is that one cheap rule, applied to queries that were already generated, can replace both round-trip filtering and extra hard-negative generation. The rule reads the relationships among samples in a structured batch and needs no additional language-model calls. If it holds, a 3B–14B open model with no filtering stage can match or beat pipelines that pay for verification. This would carry the batch-relationship idea from vision (BatchFormer, batch graphs) into the synthetic-data setting, where its main payoff is lower data-preparation cost rather than higher accuracy.

Core Claim

Training on unfiltered synthetic queries has two failure modes, and both are relational. Some generated queries are too generic or wrong to identify their source document. Others are perfectly good but also match a neighboring document, so standard contrastive training wrongly pushes that neighbor away. Both are invisible when a query is examined alone. Both become visible once the comparison set is the query's local document neighborhood rather than a random batch.

The proposal is a training recipe for a dense retriever, meaning a model that embeds queries and documents separately and ranks by dot product. It has three pieces. First, batches are built from neighborhoods in a document nearest-neighbor graph. Second, each query receives a soft reliability weight: how strongly a slowly updated copy of the retriever assigns it to its own document rather than to its neighbors. Third, each query receives a soft target over the batch documents that gives partial credit to siblings whose own synthetic queries are near-paraphrases of it.

Nothing is discarded, and the retriever is used unchanged at test time. The new costs are one nearest-neighbor graph over the documents and some extra similarity computations per batch. These are negligible next to retraining a filter retriever or doubling generation.

Concretely, a batch holds $B$ neighboring documents $d_1,\dots,d_B$, each paired with its synthetic query $q_i$. Let $p_{ij}$ be the student's softmax probability of document $d_j$ for query $q_i$. The loss is

$$ \mathcal{L} = -\sum_{i=1}^{B} w_i \sum_{j=1}^{B} t_{ij} \log p_{ij}, $$

where $w_i$ is the reliability weight and $t_{ij}$ is the soft target. The weight $w_i$ is the probability that a momentum copy of the student, initialized from the base retriever, places on $d_i$ among the $B$ neighbors, renormalized within the batch. The target is $t_{ij} \propto \mathbb{1}[i=j] + \lambda\, \max(0, \cos(q_i,q_j) - \gamma)$, which is nonzero off the diagonal only for neighbors whose queries are close paraphrases. Here $\lambda$ sets how much partial credit a sibling receives, and $\gamma$ is the similarity threshold below which a sibling gets none.

The two edge types mirror the dual-edge graph of Rethinking Batch Sample Relationships for Data Representation: A Batch-Graph Transformer Based Approach. That work combines nearest-neighbor visual edges with label-sharing edges. Here, document-neighbor edges define the batch, and query-paraphrase edges define the targets.

How This Relates to Prior Work

Two retrieval papers motivate the problem.

Study on LLMs for Promptagator-Style Dense Retriever Training shows that small open models can generate queries without filtering. It reports average BEIR NDCG@10 of 46.2 for Phi-3-medium against 46.0 for filtered Promptagator, and 48.1 with E5-base-unsupervised as the retriever. Its documents are processed independently and its training uses ordinary contrastive batches.

Promptagator: Few-shot Dense Retrieval From 8 Examples supplies the constraint that matters most here: hard round-trip filtering over-prunes on small corpora such as NFCorpus. That finding is the reason this proposal reweights instead of discarding.

Two papers buy relational signal with more generation.

[It's All Relative!

  • A Synthetic Query Generation Approach for Improving Zero-Shot Relevance Prediction](/work/its-all-relative-a-synthetic-query-generation-approach) generates a relevant and an irrelevant query per document and verifies both with the model as judge. It gains 8.6 NDCG@10 over single-query generation on five of six BEIR datasets, though its contrast is confined to one document at a time and it degrades on entity-heavy FEVER.

DISCO: Distilling Counterfactuals with Large Language Models similarly pairs generation with a teacher filter for counterfactual inference data.

This proposal asks whether the contrast these methods pay for already exists across documents, in queries that were written anyway.

UDAPDR: Unsupervised Domain Adaptation via LLM Prompting and Distillation of Rerankers absorbs synthetic noise in a different way. It trains several cross-encoder rerankers on random subsets and distills their consensus into ColBERTv2. That spends reranker training compute where this proposal spends almost none.

The mechanism borrows from the batch-relationship line of work. BatchFormer: Learning to Explore Sample Relationships for Robust Representation Learning attends across the batch during training and removes the module at test time. We keep that zero-inference-cost property but use a fixed, non-learned relational rule, for two reasons.

First, there are no clean labels to train a relational module against. Second, BatchFormerV2: Exploring Sample Relationships for Dense Representation Learning found that batch relations computed on unformed features are unstable. That finding is why reliability weights come from a momentum copy anchored at the pretrained retriever rather than from the live student.

The closest structural precedent is Exploiting Inter-sample and Inter-feature Relations in Dataset Distillation. Its class-centralization loss reached 55.1% on CIFAR-10 at 10 images per class, 6.6 points above distribution matching, in under 3,000 iterations. That paper pulls samples toward centroids. We do the opposite: we preserve legitimate overlap between neighbors and penalize only queries that fail to discriminate.

Research Program

The build is deliberately small. Queries are generated once with the Promptodile protocol: roughly eight examples per dataset and Llama-3.2-3B or Phi-3-medium as the generator, with no filtering. A nearest-neighbor graph over documents is built with the frozen base encoder, either E5-base-unsupervised or Contriever. Training samples batches as graph neighborhoods of size $B$. Each step mixes in a fraction of random documents, so the model still learns coarse discrimination.

The analysis has three parts. The first asks whether $w_i$ actually tracks query quality. It is checked against an LLM-judge label on a held-out sample and against injected corruptions, namely queries swapped to random documents and queries truncated to generic stems. The second asks whether the soft targets $t_{ij}$ fire on true false negatives. Using BEIR's relevance judgments, we measure how often a sibling given credit is in fact judged relevant to the query. The third tracks how the benefit varies with neighborhood size $B$ and corpus size. Small specialized corpora like NFCorpus and SciFact are where filtering over-prunes, so they are where reweighting should matter most.

The cost accounting is part of the result. Every method's data-preparation budget is reported in language-model tokens generated and GPU-hours, alongside retrieval quality.

Baselines

The naive reference point is Promptodile exactly as published: unfiltered queries from the same generator, the same base retriever, and random in-batch negatives. Its reported averages are 46.2 NDCG@10 for Phi-3-medium with Contriever-style training and 48.1 with E5-base-unsupervised, from Study on LLMs for Promptagator-Style Dense Retriever Training. We rerun it ourselves on the same query files and the same step count, so the comparison does not depend on reproduction drift.

The strongest cost-matched competitor is the published Promptagator recipe: a temporary retriever trained on the queries, with hard removal of any query that fails to retrieve its source in the top ranks. It is the same thing $w_i$ tries to do softly. Its reported filtered score is 46.0 NDCG@10 in the Promptodile comparison. We run it on our queries with the same retriever and count its extra training run in the cost ledger.

The strongest quality competitor is GENERATE-PAIRWISE, re-implemented with our small generator so that model size is held fixed. We give it its extra irrelevant-query generation and judge pass, and report quality at both equal training steps and equal generation tokens. Its published gain is 8.6 NDCG@10 over single-query generation on five of six BEIR datasets, with an mT5-XXL ranker. That setup is not directly comparable, so our re-implementation is the real reference.

UDAPDR is a secondary comparison on LoTTE, where it reports Success@5 rising from 64.5 to 74.8 on Lifestyle. We run its multi-reranker distillation with the same generated queries but a single-vector student. This shows what reranker compute buys over a free relational rule.

Two ablations isolate the key ingredient: neighborhood batches with ordinary one-hot targets and no weights, and neighborhood batches with weights only or targets only. Beating the method means two things. Average NDCG@10 over the 11 Promptagator BEIR datasets must exceed both rerun Promptodile and filtered Promptagator, with paired bootstrap significance. It must also come within the noise of re-implemented GENERATE-PAIRWISE while using roughly half its generation tokens and no judge pass.

Why not just…?

Why not just drop filtering, as Promptodile shows you can? That is the do-nothing baseline, and it is a real threat on large corpora, where noise averages out. The proposal predicts its gains concentrate on small specialized corpora such as NFCorpus and SciFact, and on datasets with many near-duplicate documents, where false negatives are common. If the relational loss does no better than plain unfiltered training on those subsets, the idea is wrong.

Why not just mine hard negatives from neighboring documents? This is the most dangerous alternative, because neighborhood batching by itself is standard hard-negative mining. The first ablation runs it directly. Our prediction is that plain neighborhood batches help on some datasets but hurt on duplicate-heavy collections, because they push away true positives. If the soft targets do not reverse that harm, as measured by the false-negative rate against BEIR relevance judgments, then the contribution reduces to hard-negative mining. We would report it as such.

Why not just filter online with the student's own loss, keeping only small-loss pairs? This is cheap and close in spirit to $w_i$. The difference is the comparison set and the decision type. A per-sample loss against random negatives cannot tell a generic query from a good one, because both beat random documents easily. Hard thresholds also reproduce the over-pruning Promptagator saw on NFCorpus. We run a small-loss variant with the same momentum encoder under both random and neighborhood batches. If it matches our method, the relational framing adds nothing beyond the batch construction.

Why not just pay for pairwise generation, since its gains are large? If the budget allows, that may be the right call for maximum quality. The question here is how much of that gain survives without the second generation call and the judge. GENERATE-PAIRWISE also degrades on FEVER-style entity retrieval, while cross-document siblings in entity-dense corpora naturally differ by entity. We expect the relational approach to hold up better on FEVER specifically, and that is a separate check.

Proposed Experiments

The main experiment covers the 11 BEIR datasets used by Promptagator, evaluated by NDCG@10. Each method is crossed with two generators (Llama-3.2-3B and Phi-3-medium) and two base retrievers (Contriever and E5-base-unsupervised), with three seeds. Every result carries its preparation cost in tokens and GPU-hours. LoTTE Success@5 is the out-of-benchmark check against UDAPDR's setting.

The decisive checks are four.

The first is a corruption experiment. We replace 10%, 25%, and 40% of queries with mismatched or generic ones, then measure two things: the AUC of $w_i$ at ranking the corrupted queries last, and how retrieval quality degrades compared with hard round-trip filtering. Soft weights should degrade gracefully where hard filtering either misses the noise or prunes good data.

The second is the NFCorpus over-pruning test. It compares the number of retained effective training pairs and the resulting NDCG@10 across hard filtering, small-loss filtering, and the proposed weights.

The third is the false-negative audit of $t_{ij}$ against human relevance judgments.

The fourth is a matched-token comparison with GENERATE-PAIRWISE, including FEVER and Climate-FEVER, separately reported.

The idea is supported only if it beats rerun Promptodile and filtered Promptagator on average, approaches pairwise generation at about half the generation cost, and shows its gains where the relational account predicts them. If the gains are uniform across datasets, or vanish once neighborhood hard negatives are controlled for, the relational account is not what explains them.