One Query, Many Soft Positives: Set-Level Labels for Cheaper Cross-Lingual Synthetic Training Data

Training a search model for a language with no labeled queries now usually works like this: a large language model reads each document and writes a plausible query for it, and the resulting (query, document) pairs become training data. Every generated query costs GPU time or API fees. Each one also yields exactly one positive label, and pipelines often throw away a large share of queries in verification passes. We propose to get more supervision out of each paid-for generation. The generator is shown a small set of related documents and writes one query. A cheap probabilistic scorer then spreads that query's relevance across the whole set as a soft distribution, instead of stamping a single document "relevant" and the rest "irrelevant."

The idea borrows a lesson from older cross-lingual retrieval and parsing work. When the mapping between languages is ambiguous, keeping a weighted distribution over candidates beats committing to one. We apply that lesson to the bottleneck of synthetic data: generation cost. If one query labeled softly against a set of $k$ documents carries close to the information of $k$ separately generated queries, the cost of building a cross-lingual training set could fall several-fold without any extra filtering stage.

Core Claim

In cross-language retrieval (English queries, non-English documents), a training signal built from set-level soft labels can match per-document synthetic generation at a fraction of the generation budget. It should also beat per-document generation when the budget is held fixed. The method has three steps. First, cluster target-language passages into small groups. Second, prompt a small open model to write one English query per group, aimed at a designated anchor passage. Third, score that query against every group member with the generator's own query likelihood, normalized over the set, and train the retriever to match the resulting distribution.

What is new is the unit of labeling. Prior soft-label methods give soft scores to queries that were still generated one per document, using a teacher trained on human labels. Here the soft distribution is defined over a set chosen before generation, and it is scored without a gold-trained teacher. That choice is what lets it reduce the number of generations rather than only the cost of filtering them.

Where this sits relative to prior work

Two lines of work motivate the proposal.

The first is the literature on probabilistic matching across languages. It showed that keeping weighted alternatives beats hard selection. Nie et al. folded translation probabilities directly into retrieval scoring and found that translation ambiguity acts as useful query expansion. Xu and Weischedel summed over all lexicon translations inside a hidden Markov retrieval model. CLGVSM linked documents across languages through translation probability matrices and beat hard matching. SubDP is the closest methodological template: it projected soft distributions over dependency arcs through stochastic alignments and trained parsers with cross-entropy against those "soft silver labels." This raised Hindi labeled attachment score from 28.3 to 38.9, and it worked with as few as 50 bitext pairs. We reuse SubDP's recipe of supervising with a projected distribution rather than a hard choice. Unlike SubDP, our distributions are over retrieval relevance within a document set, and the data comes from a language model rather than an existing treebank.

The second line is synthetic retrieval data, whose overhead motivates the cost side. Promptagator generated millions of queries with a 137B model and filtered them by round-trip consistency, which was too strict on small corpora such as NFCorpus. GENERATE-PAIRWISE generated relevant and irrelevant queries together and added a language-model judging step, whose authors acknowledge the data-preparation overhead. UDAPDR replaced filtering with distillation from an ensemble of large rerankers, which is expensive in its own right.

Three papers constrain the design. JH-POLO is the direct cross-lingual predecessor. It prompted GPT-3 with positive and negative non-English passages, kept hard labels, and discarded queries that failed a cross-encoder check, at about $400 for one Chinese training set. We keep its multi-passage prompting but replace its hard triple check with a soft set distribution, and we retain every query. Noisy Self-Training is the closest existing soft-label work. It scores synthetic queries against a passage pool with a teacher retriever trained on MS MARCO gold labels and distills those soft scores. It is monolingual English, generates per passage, and depends on a gold-trained teacher. Our scorer needs none of these, and our sets are built before generation. Promptodile is the most uncomfortable result for any noise-handling method. Unfiltered 3B and 14B open models reached 45.1 and 46.2 nDCG@10 on BEIR, matching or beating filtered 137B Promptagator (46.0). That result tells us not to sell a better filter. It is why our claim is about budget, not about noise removal.

Research Program

Building passage sets. Target-language passages from each corpus are grouped into sets of size $k \in \{1, 2, 4, 8\}$ using cheap multilingual embedding similarity. Setting $k=1$ recovers ordinary per-passage generation. One passage per set is the anchor. The others act as near-neighbors, playing the role of the plausible alternative translations in the probabilistic retrieval literature.

Generation. An open model from the Promptodile size range (Llama-3.2-3B and Phi-3-medium) receives the set, with the anchor marked, and writes one English query. Three prompt variants are compared: a query specific to the anchor, a query shared across the set, and the JH-POLO style of specific-versus-contrast passages.

Set-level scoring. For a query $q$ and set $S=\{d_1,\dots,d_k\}$, the soft label is

$$ y_i = \frac{\exp\left(\log p_\theta(q \mid d_i)/\tau\right)}{\sum_{j=1}^{k} \exp\left(\log p_\theta(q \mid d_j)/\tau\right)}, $$

where $p_\theta(q \mid d_i)$ is the generator's length-normalized likelihood of the query given passage $d_i$, and $\tau$ is a temperature. This needs only $k$ prefill passes, with no decoding. As a teacher-free second scorer we use a translation-probability term match in the style of Nie et al., with translation tables estimated from a small bitext. Following SubDP's few-bitext result, we vary the bitext size from 50 to 25,000 pairs.

Training. A multilingual dual encoder of the DPR-X type used by JH-POLO is trained with KL divergence between its softmax over the set plus in-batch negatives and the target $y$. Documents outside the set receive near-zero mass.

Cost accounting. Every arm reports GPU-hours, split into decoding, prefill scoring and training. At an assumed GPU price we convert to a dollar figure so arms can be compared against JH-POLO's reported API cost.

The main risk is false positives. A near-neighbor can receive probability mass for a query it does not answer, the alignment-noise failure that the probabilistic literature flags. The temperature and the anchor-specific prompt are the controls for this, and the ablations below measure the risk directly.

Baselines

The naive reference is per-passage generation with the same open model, no filtering, and a hard single positive. This is the Promptodile recipe transplanted to cross-lingual retrieval, and it is the $k=1$ setting of our system. It is the strongest cheap competitor, because Promptodile showed that unfiltered small-model data is already competitive on BEIR. It is run with identical passages, generator, retriever architecture and training steps. It is compared at equal generation budget, measured in decoded tokens and GPU-hours, and also at $k$ times our budget. Beating it means higher nDCG@20 at equal budget, or statistical parity at no more than half the budget.

The strongest existing cross-lingual method is JH-POLO: contrastive prompting followed by a hard cross-encoder filter. It reports 0.269 nDCG@20 on HC3 Persian tweets, against 0.177 for an mMARCO-trained model. For NeuCLIR 2022 no per-language reported result was found on Lacuna, so we rerun it. To keep the generator from confounding the comparison, we re-implement JH-POLO's filter on the same open generator, and we also cite its GPT-3 number as an external reference. Beating it means matching 0.269 on HC3 Persian at lower measured cost, or exceeding it at equal cost.

The strongest soft-label competitor is a Noisy Self-Training-style teacher. A multilingual retriever trained on machine-translated MS MARCO scores per-passage synthetic queries against a pool, and the student distills those scores. It gets the same student, the same number of training steps and the same tuning budget of 10 trials. It has access to gold-derived supervision that our method lacks, so we expect it to be strong, and our claim against it is about cost.

A plain machine-translated MS MARCO model (mMARCO) and BM25 over translated queries serve as zero-generation reference points.

Three ablations isolate the key ingredient. The first keeps the set prompt but labels only the anchor, testing whether soft labels or merely richer prompts drive any gain. The second spreads uniform labels over the set, testing whether the scoring matters or multi-positive training alone suffices. The third uses random sets instead of similarity clusters, testing whether set coherence matters.

Why not just…?

Why not just generate unfiltered queries with a small model, as Promptodile did? This is the real threat. If per-passage generation at $k$ times the budget only matches our method, while at equal budget the $k=1$ arm wins, the method has failed. The equal-budget and multiple-budget comparisons against the $k=1$ arm settle this directly. Our hypothesis rests on two observations. Near-neighbor passages are where hard labels mislead most, a point Noisy Self-Training makes about spurious correlations under hard labels. And neighboring documents in a distant language are exactly where the probabilistic retrieval literature found weighting pays.

Why not just use a gold-trained multilingual teacher to soft-label everything? It may well give better labels. But it presupposes a human-labeled training set rendered into the target language, which JH-POLO shows transfers poorly to informal domains: the mMARCO model reached 0.177 against 0.269 on Persian tweets. The teacher baseline above measures how much quality our teacher-free scorer gives up, both in-domain (NeuCLIR news) and out of domain (HC3 tweets). We will report it honestly even if it wins in the news domain.

Why not just machine-translate MS MARCO? On NeuCLIR news JH-POLO found this roughly as good as synthetic data, so a tie there would not be surprising. The informal HC3 setting is where the comparison is decisive, and mMARCO is included as a reference arm.

Proposed Experiments

The main experiment runs on NeuCLIR 2022 (Chinese, Persian, Russian news) and HC3 (Persian and Chinese tweets), with nDCG@20 as the metric. It sweeps $k \in \{1,2,4,8\}$ and total generation budget across a logarithmic grid, giving quality-versus-GPU-hour curves for every arm. The decisive check is whether the $k=4$ curve lies above the $k=1$ curve across the grid, and whether it reaches JH-POLO's 0.269 on HC3 Persian at a measured cost well below the roughly $400 JH-POLO reports for a single language.

A second check measures label quality directly. On a few hundred human-judged (query, set) pairs per language we compute the calibration of $y$ and the false-positive mass placed on non-relevant neighbors, and we relate it to downstream gains as $\tau$ and $k$ vary. We expect gains to peak at moderate $k$, where false-positive mass starts to rise. A sweep of bitext size for the translation-probability scorer tests whether the SubDP few-bitext pattern holds, meaning near-full performance from 50 pairs.

Finally, a monolingual sanity check on BEIR compares against Promptodile's 45.1 (3B) and 46.2 (14B) nDCG@10 and Promptagator's 46.0 filtered and 44.7 unfiltered. It focuses on NFCorpus and SciFact, where Promptagator's filter over-pruned. It tells us whether set-level labeling is specifically cross-lingual or helps whenever corpora are small and documents crowd together.