Entity-Anchored Query Synthesis for Low-Resource Dense Retrieval

Training a dense retriever (a search model that maps questions and passages to vectors and retrieves by vector similarity) for a language like Swahili or Telugu usually means asking a large language model to invent millions of plausible questions for the passages in that language, then paying again to throw away the bad ones. The bad ones are common. The generator hallucinates entities, drifts into English, or writes generic questions any passage could answer. The standard fix is a second model that checks each question, which roughly doubles the bill. We propose making the question generator cheaper to trust. Every Wikipedia passage in the target language already contains hyperlinks that resolve, through inter-language links, to language-independent Wikidata identifiers. If each synthetic question is built around a specific pair of those identifiers, then checking the question reduces to a dictionary lookup: does it mention the anchored entities, and no unlinked or foreign ones? The same entity graph also supplies hard negative passages, meaning passages that share one anchor but not the other, without running any reranker.

The claim is about cost as much as accuracy. At a fixed generation budget, anchoring questions to knowledge-base identifiers should waste fewer generated samples and replace a learned consistency filter with an almost free symbolic one. It should also produce a student retriever at least as good as one trained with the usual generate-then-filter pipeline. If this holds, low-resource dense retrieval becomes affordable for labs without access to hundred-billion-parameter teachers. It also turns pivot-based entity mapping, a tool previously used for recognition and linking, into a control mechanism for synthetic data.

Core Claim

The claim has three parts. First, conditioning question generation on Wikidata-anchored entity pairs from the source passage lowers the fraction of generated questions that must be discarded. Second, an identifier-agreement check can stand in for round-trip retriever filtering with no loss in the final retriever's quality. Third, knowledge-base-derived hard negatives can replace cross-encoder teachers. Together these should move the curve of retrieval quality against generator compute on the Mr. TyDi benchmark. The novelty is not using entities for augmentation, which LEIA does. It is using identifiers as both the generation constraint and the verification signal in a retrieval-data pipeline, and measuring what that saves.

Positioning

The research file supplies two sides. On the synthetic-data side, Promptagator prompts a 137B model with eight examples to write queries over a corpus. It then trains a temporary retriever and discards queries that fail to retrieve their own source passage. This round-trip filter is the main cost we target. The same paper notes that on small corpora the filter throws away too much data, which is a direct motivation for a filter that does not depend on corpus size. UDAPDR avoids explicit filtering by generating with a cheaper model and distilling from a family of DeBERTa cross-encoders into ColBERTv2. That trades filter cost for teacher-training cost, and we test whether entity-graph negatives can remove that cost too.

Balancing Cost and Effectiveness of Synthetic Data Generation Strategies for LLMs plays two roles here. It supplies the budget-ratio lens we use to compare pipelines at equal query budgets. It also reports that verifying synthetic answers against ground truth did not significantly help students. That finding constrains our proposal: if verification is useless in our setting too, then the identifier check is pointless, and only the conditioning half of the idea can matter. We treat this as a live hypothesis, not a straw man.

On the pivot side, LEIA supplies the mechanism. It maps target-language Wikipedia hyperlinks to English titles via inter-language links and injects them as anchors during language adaptation. We reuse the same deterministic extraction, but for building and checking task data rather than for continued pre-training. Soft Gazetteers shows that continuous features from linking low-resource n-grams to English Wikipedia help NER. It also warns that such features are bounded by the linker's recall and fail on entities missing from the knowledge base. That is exactly the failure mode our verifier will have, so we measure it. PLUG reaches strong data efficiency by generating pivot-language intermediate text at inference. It motivates keeping the pivot offline: our student retriever sees only target-language text at query time.

The closest cross-lingual synthetic pipeline is Synthetic Data Augmentation for Zero-Shot Cross-Lingual QA. It generates questions over multilingual Wikipedia, either in English or through machine-translated generators, and distills them into MiniLM. It relies on translation and unconstrained generation, and it reports a 0.65 correlation between question quality and downstream QA. That correlation motivates spending effort on question quality, and we propose to buy that quality with identifiers rather than with translation. Translate & Fill similarly replaces brittle alignment heuristics with generative span filling, but it still needs upstream machine translation. Our pipeline needs none.

Research Program

The pipeline runs over the passage collections of Mr. TyDi, which are target-language Wikipedia, so hyperlinks are native.

For each passage, we extract hyperlinked mentions and resolve them to Wikidata identifiers. We keep, per identifier, its target-language aliases and anchor texts. When two anchored entities in a passage are joined by a Wikidata relation, we record the relation label. We then sample an anchor bundle: one or two identifiers, their target-language surface forms, and the optional relation. We prompt a moderately sized open multilingual generator, deliberately far smaller than Promptagator's teacher, to write a target-language question that the passage answers and that names the bundled entities.

Verification is symbolic. A question is kept if two conditions hold. It must contain an alias of each anchored identifier. It must also contain no span that a cheap anchor-text dictionary links to an identifier absent from the passage's anchor set, since such a span is a likely hallucinated entity. A script check rejects outputs that drift out of the target script.

For hard negatives, we use passages that share one anchored identifier but lack the other, or that contain the head entity with a different relation. These give negatives that are topically close but wrong, at no model cost. The student is a multilingual dual encoder of the same size as mDPR, trained with contrastive loss on anchored questions, graph negatives, and in-batch negatives.

The analysis has three threads. The first is a cost accounting. We log generator tokens, filter compute, and teacher compute per retained training pair, for every pipeline, and plot final retrieval quality against total GPU-hours. The second is a failure analysis on questions whose gold answer involves entities the knowledge base lacks, which covers local, emerging, and NIL entities. There we measure how much recall the verifier loses, following the linker-recall concern in Soft Gazetteers. The third is a coverage analysis. Anchored questions skew toward entity-centric information needs, so we measure the student's performance separately on Mr. TyDi test questions with and without linkable entities.

Baselines

The naive reference points are tuned BM25 and zero-shot mDPR, both as reported in Mr. TyDi. On Swahili, that paper reports MRR@100 of 0.396 for tuned BM25 against 0.073 for zero-shot mDPR. It also reports that a BM25 plus mDPR hybrid beats BM25 in 9 of 11 languages, with an average MRR@100 gain of 8 points. We rerun both, plus the hybrid, on our splits. Any synthetic pipeline that does not beat tuned BM25 in the lowest-resource languages has failed outright.

The strongest existing method for this question is Promptagator-style generation with round-trip filtering, reimplemented in the target language. We use the same open generator, the same eight in-language few-shot examples from Mr. TyDi training data, the same passage set, the same student architecture, and the same hyperparameter search budget. We do not scale the teacher to 137B; both pipelines share one generator so that the only difference is the pipeline. No reported result found on Lacuna for Promptagator-style synthesis on Mr. TyDi. Its reported BEIR figure is 47.8 average nDCG@10 in the few-shot setting, but that is English and not comparable. To beat this baseline, anchored synthesis must match or exceed its MRR@100 and Recall@100 at equal generator-token budget while using clearly less total compute, or it must exceed its quality at equal total compute. Merely matching quality at equal total compute is a null result.

The second strong competitor is a UDAPDR-style pipeline. It uses cheap unfiltered generation, cross-encoder teachers trained on data slices, and distillation, following UDAPDR, which reports Success@5 on LoTTE Lifestyle rising from 64.5 to 74.8. We give it the same generator tokens and charge its cross-encoder training and scoring to its compute account. Graph negatives win if they match its student quality without that cost.

Three ablations isolate the ingredients: anchor conditioning with no verification, unanchored generation with the identifier check applied afterward, and the full method with in-batch negatives only. Together they separate what conditioning, the symbolic filter, and graph negatives each contribute. A final zero-LLM baseline instantiates Wikidata relation templates as target-language questions directly. It measures how much the generator adds beyond the knowledge base itself.

Why not just…?

Why not just translate an English retrieval set into the target language? Translate-train is cheap to set up, but it needs a machine translation system of usable quality. For languages like Swahili or Telugu, that system is exactly the missing resource. The QA augmentation study found that translating into English as a pivot was beaten by in-language synthetic data because translation propagated errors and lost nuance (Synthetic Data Augmentation for Zero-Shot Cross-Lingual QA). Translated queries are also not grounded in the target collection's passages. We nonetheless include a translate-train student, built from translated MS MARCO-style queries with an open translation model, at matched compute. Where good translation exists, it may well be competitive, and the per-language results will show that.

Why not just generate far more unfiltered questions with a cheap model? This is the real threat, because Balancing Cost and Effectiveness found that verification gave little benefit. Against it, DISCO found that 75,000 filtered examples beat over a million noisy ones. The conditioning-only ablation and a volume-scaled unfiltered baseline, which uses the full method's total compute to generate more questions, settle this directly. If unfiltered volume matches the full method, we drop the verifier and report that anchoring's value, if any, lies in conditioning alone.

Why not just skip the language model and build questions from Wikidata templates? Templates are free and perfectly verifiable. They are also stylistically unlike real user questions and limited to relations the knowledge base encodes. The zero-LLM template baseline measures this gap. If templates come close, the honest conclusion is that the knowledge base, not generation, does most of the work, which would itself be a useful result for compute-poor labs.

Why not just use the BM25 plus mDPR hybrid? Mr. TyDi shows the hybrid helps, but its gains are capped by a dense component that scores 0.073 MRR@100 on Swahili. We report the hybrid, and we also report the hybrid built with our student in place of mDPR, to test whether the gains add up.

Proposed Experiments

The main experiment trains students on the Mr. TyDi languages, prioritizing the low-resource ones such as Swahili, Bengali, and Telugu, under generator budgets that span a low-to-high budget-ratio range in the sense of the Balancing Cost paper. We report MRR@100 and Recall@100 on the Mr. TyDi test sets, as a function of both generator tokens and total GPU-hours, for every pipeline above.

The first decisive check concerns efficiency. Anchored synthesis should show a lower discard rate than round-trip filtering, and a quality-versus-compute curve that sits above the Promptagator-style curve in at least the low-resource languages. If the curves overlap, the idea saves nothing.

The second check concerns the verifier. We manually annotate 300 retained and 300 rejected questions per language for answerability and entity hallucination. From this we estimate the identifier check's precision and recall relative to human judgment and to the round-trip filter.

The third check concerns coverage. We compare student performance on test questions whose answers involve knowledge-base-linked entities against those that do not. A large drop on the second group would show that entity anchoring buys cheap verification at the price of narrower coverage. That would be reported as a limit of the approach, with a mixed pipeline, anchored plus a small unanchored share, tested as the remedy.