Does Synthetic Evaluation Overstate Synthetic Document Expansion? A Per-Language Audit

Search engines often fail because people describe what they want in different words than the documents use. One cheap fix is document expansion: before indexing, a text generator writes the questions each document might answer and appends them to it, so a plain keyword index can match more user phrasings. This works in English. Extending it to the hundreds of languages that lack human-written test queries creates a trap. The natural way to check whether expansion helps in such a language is to have a language model write test queries too. But the expansion text and the test queries then come from the same kind of generator, with the same clean phrasing and the same word choices. The evaluation may be measuring how well generated text matches other generated text, not how well real users are served.

We propose to measure this inflation directly and per language. We use MIRACL, a benchmark of human-written queries and human relevance labels in 18 languages, where both a synthetic and a human evaluation of the same expanded index can be run side by side. We then test whether cheap, label-free language properties can predict how much synthetic evaluation overstates expansion gains. If they can, the predictor gives a correction for languages where no human evaluation exists. If they cannot, that is a concrete warning against building the next generation of multilingual retrieval benchmarks the way some already are.

Core Claim

Our hypothesis is that, when a corpus is expanded with generated queries, the measured retrieval gain on generated test queries exceeds the gain on human queries. We further expect the excess to differ across languages in a way that tracks how cleanly the generator and tokenizer handle each language. We will report two quantities per language $\ell$. The first is the inflation ratio $I_\ell = \Delta^{syn}_\ell / \Delta^{hum}_\ell$, where $\Delta$ is the change in nDCG@10 (a rank-weighted relevance score) from adding expansion, measured on synthetic test queries ("syn") or human test queries ("hum"). The second is an exchange rate $k^*_\ell$: the number of generated queries per document needed to match the gain from attaching one real human query to that document. The exchange rate borrows the substitutability idea that On the Equivalency, Substitutability, and Flexibility of Synthetic Data developed for video tracking. That paper replaced real training frames with simulated ones and found roughly 30 synthetic frames were worth one real one. We carry the same question into text retrieval and across languages, where nobody has measured it.

Where This Sits

Three lines of work supply the pieces, and none of them combines expansion with an audit of synthetic evaluation.

The expansion method comes from Document Expansion by Query Prediction. That paper, called Doc2query, trains a sequence-to-sequence model to write about ten queries per English passage and appends them before keyword indexing with BM25, the standard term-weighting ranker. It raised MS MARCO MRR@10 (the average reciprocal rank of the first relevant passage among the top ten) from 18.4 to 21.5, and roughly 31% of the generated words were absent from the passage. Doc2query was evaluated only in English and only on human queries. We reuse its method and change the evaluation.

Two papers motivate the concern about real versus generated queries. Click-based expansion with real queries, through ORCAS and Neural Document Expansion with User Feedback, shows that real user queries attached to documents carry strong signal. ORCAS reports MRR moving from 0.902 to 0.931. NeuDEF reports gains of 20.9% to 28.1% in MRR on Chinese logs. Such logs exist only inside large search engines, which is why generated queries are the realistic substitute in most languages.

The evaluation lens comes from Synthetic Test Collections for Retrieval Evaluation. It showed that in English, synthetic queries with LLM relevance judgments rank retrieval systems much as human collections do (Kendall's $\tau$ = 0.8568), while absolute scores are consistently higher, meaning synthetic collections are easier. It also found little bias toward systems sharing the generator's model.

That last finding appears to contradict our hypothesis, so we address it directly. That study compared retrieval systems that never had generated text inside their index. In expansion, generated text sits in the index itself and can match generated test queries word for word. That is a far more direct path to bias than a shared model architecture. The concern is not hypothetical. MMTEB already includes GPT-4-generated multilingual Wikipedia retrieval tasks as evaluation. mMARCO uses machine-translated queries and showed that the wording mismatches translation introduces hurt BM25 specifically. Both are evaluation sources whose word choices differ systematically from those of real users.

Making a MIRACL provides the ground truth. Native speakers wrote about 77k queries and over 700k relevance judgments across 18 languages. BM25 quality varies widely across them, for example 0.183 nDCG@10 in French against 0.551 in Finnish.

Research Program

We work on the MIRACL Wikipedia corpora.

Expansion indexes. For each language we build three indexes over the same passages. The first is unexpanded. The second is generator-expanded, with $k \in \{1, 2, 5, 10, 20\}$ generated queries per passage written directly in the target language. We use two generators: a fine-tuned multilingual sequence-to-sequence model in the Doc2query style, and an instruction-tuned LLM. The third is real-query-expanded: we attach MIRACL training-split human queries to the passages they were judged relevant to, the human-written analogue of ORCAS click metadata. Training and evaluation queries are disjoint, so this expansion is legitimate metadata rather than leakage.

Human queries cover only a subset of passages, so we compute the exchange rate on that covered subset. On the same subset we compare the generator-expanded index against the real-query-expanded index.

Evaluation regimes. Each index is scored under four evaluation regimes:

  • Human: MIRACL dev queries with their human judgments.
  • Same-generator synthetic: synthetic test queries from the expansion generator, judged by an LLM over pooled candidates. We use full pooled judgments because Synthetic Test Collections showed that treating only the source passage as relevant collapses agreement to $\tau$ = 0.157.
  • Cross-generator synthetic: the same procedure, but test queries come from a different model family than the expansion generator.
  • Translated: human English queries machine-translated into the target language, in the mMARCO style.

We validate the LLM relevance judge per language against MIRACL's human judgments before trusting it.

Predictor. We fit a deliberately small regression from label-free language features to $\log I_\ell$, the logarithm of the inflation ratio. Small means few features and strong regularization, because there are only 18 training points. The features are:

We use leave-one-language-out error, and we hold out MIRACL's two surprise languages as a final test the predictor never sees during design.

Baselines

The naive reference point is the unexpanded BM25 index evaluated on human queries. MIRACL reports per-language BM25 figures, such as 0.183 nDCG@10 for French and 0.551 for Finnish, and we reproduce these before anything else. Any expansion claim is a change relative to this index under identical tokenization and BM25 parameters.

The strongest system-level competitor is the hybrid of BM25 and mDPR, a multilingual dense retriever that encodes text as vectors. Making a MIRACL reports an average of 0.578 nDCG@10 and 0.889 Recall@100 for this hybrid. We rerun it with the sparse half swapped for each expanded index, keeping fusion weights tuned on the same dev slice for every condition. This tells us whether any expansion gain survives once dense matching is present.

We also include multilingual-e5-large-instruct, which MMTEB names as the best public model on its benchmark. No reported MIRACL result was found on Lacuna for it, so we measure it ourselves.

Search-time query expansion with RM3 pseudo-relevance feedback serves as the non-generative alternative. Doc2query beat it in English. We tune its feedback depth on the same dev budget as the expansion depth $k$.

Two ablations isolate the key ingredient. The first is the real-query-expanded index, which separates "more text appended" from "generated text appended." The second matches the token count of generated and real expansions, so that any difference cannot come from length alone.

For the evaluation question itself, the baseline is the English result: synthetic-versus-human system ordering agreement of $\tau$ = 0.8151 for synthetic queries with human judgments and 0.8568 for fully synthetic collections, both from Synthetic Test Collections. We would count the inflation finding as established if three conditions hold:

  • same-generator synthetic evaluation gives $I_\ell$ reliably above 1 across languages, by bootstrap over queries;
  • its $\tau$ against human evaluation, computed over our grid of expansion variants, falls clearly below the English figures;
  • the predictor's leave-one-language-out error beats the trivial predictor that assigns every language the mean inflation.

Why not just…?

Why not just use dense multilingual retrieval and forget about keyword expansion? Dense retrieval does not remove the problem. MIRACL's best configuration was still a dense–sparse hybrid, so the sparse half matters. Arctic-Embed 2.0 also reports limited cross-lingual transfer outside the pretraining languages. More importantly, the evaluation question applies to any system tuned or judged on generated queries. Our hybrid runs will show whether expansion still adds anything once dense matching is in place. If it adds nothing, we will report that, and the inflation audit remains relevant to dense systems evaluated on synthetic benchmarks.

Why not just use a different generator for the test queries than for the expansion? This is the real threat. If cross-generator evaluation alone brings $I_\ell$ near 1 and restores high $\tau$, then the fix is cheap and the predictor is unnecessary. We would publish that as the main result: a concrete protocol rule for multilingual synthetic benchmarks. The cross-generator arm exists to settle this. Our expectation that it is insufficient rests on the "easier collections" finding, which suggests regularized phrasing is shared across LLM families rather than specific to one model, but we treat this as a hypothesis, not a premise.

Why not just evaluate with translated English human queries? Translated queries keep human intent, but mMARCO showed that translation alters word choice in ways that specifically mislead keyword matching. That is exactly the effect expansion is supposed to address, so translated evaluation may understate or overstate gains in its own direction. We include it as an evaluation regime and report its $I_\ell$ alongside the synthetic regimes.

Why not just measure on MIRACL's 18 languages and assume the rest behave like the average? The predictor exists to test whether that assumption is worse than using language features. The mean-inflation predictor is our explicit baseline. If it wins on the surprise languages, we will conclude that per-language correction is not yet reliable with this many languages, and say so.

Decisive Checks

Three outcomes decide the proposal.

The first is whether same-generator synthetic evaluation inflates expansion gains on human-verified ground truth, and whether that inflation varies by language beyond bootstrap noise. If $I_\ell \approx 1$ everywhere, the concern is unfounded, which is a useful negative result for benchmarks like MMTEB's synthetic retrieval tasks.

The second is whether decoupling the generators removes the inflation.

The third is whether the exchange rate $k^*_\ell$ is finite and stable. A finite rate would let practitioners in languages without click logs budget generated expansions against the value of real queries. An infinite rate, where no number of generated queries matches one real query, would mean generated expansion cannot replace real query data in some languages. We expect the two to differ most where the local-structure probe or tokenizer fertility flags a language as poorly handled. The two held-out surprise languages give a single, pre-registered test of that expectation.