Setting Hybrid Retrieval Weights for Languages Without Relevance Judgments

Hybrid search combines two retrievers. A lexical one (BM25, which matches exact words) and a dense one (a neural encoder that maps queries and documents to vectors and matches by similarity) each score every candidate document, and a weighted sum of the two scores decides the ranking. On multilingual benchmarks this combination is reliably better than either part alone. The weight that balances the two, however, is tuned on relevance judgments, and those exist for only a few dozen of the hundreds of languages that multilingual encoders are trained on. In every other language a deployed system runs on a guessed weight.

We propose to predict that weight, per language and per query, from signals that need no judgments at all. The first is how badly the dense model's tokenizer fragments the query's words; heavy fragmentation is the symptom of a vocabulary-coverage gap. The second is a cheap probe of whether the dense model understands the language at all. The bet is that vocabulary coverage explains much of why the best weight differs so much across languages and queries. If it does, a small predictor trained on languages that have judgments should carry over to languages that have none.

Core Claim

Two published results suggest the best weight between lexical and dense scores varies a great deal. Making a MIRACL builds an 18-language Wikipedia retrieval benchmark with native relevance judgments. It reports that BM25 alone ranges from nDCG@10 of 0.183 in French to 0.551 in Finnish, while a fixed linear hybrid of BM25 and mDPR (a multilingual dense retriever built on mBERT) averages 0.578 nDCG@10 and 0.889 Recall@100. Mr. TyDi transfers an English-trained mDPR zero-shot to 11 languages. There dense retrieval collapses in Swahili (MRR@100 of 0.073 against 0.396 for BM25), yet the hybrid still beats tuned BM25 in 9 of 11 languages, by an average of 8 MRR@100 points. Both papers use one fixed combination. In English, Out-of-Domain Semantics to the Rescue! finds that whether dense or lexical signals dominate shifts with query length and vocabulary shift, and names query-dependent weighting as open.

Our claim is that a large share of this variation can be predicted without judgments from vocabulary-coverage features. The main query-level feature is the fraction of query words that the dense tokenizer splits into many pieces, a per-query version of the "fertility" (subwords per word) statistic. The main language-level features are corpus fertility and an unsupervised intelligibility score. A predictor fit on judged languages should then recover most of the gain that per-language tuning on judgments would give, on languages it never saw.

What is new is the target and the constraint. Prior multilingual hybrids fix the weight. Prior performance predictors estimate task accuracy rather than a control decision. Prior vocabulary-gap work repairs embeddings rather than routing between retrievers.

Research Program

The vocabulary-coverage direction supplies the features. Multi Task Learning For Zero Shot Performance Prediction regresses cross-lingual task accuracy on typology, corpus size, subword overlap and tokenizer fertility. Its attribution analysis finds fertility a leading driver of degradation on word-level tasks. We reuse that insight but move it from language level to query level, and point it at a fusion weight instead of an accuracy score. LOVE shows that subword tokenizers break on rare and corrupted words, and imputes replacement embeddings with a small contrastive encoder, but only for English classification and named-entity recognition. It motivates our premise that fragmented words are where dense representations fail. We do not adopt its repair as the main mechanism.

The evaluation-gap direction supplies the regime and a gate. Detecting Languages Unintelligible to Multilingual Models proposes a label-free probe that swaps adjacent characters in raw sentences. The probe measures how well a model still matches originals to their scrambled versions. It reports a correlation of r = 0.99 with XNLI and Tatoeba accuracy across 44 languages, and a 0.99 threshold below which downstream accuracy is at chance. That result constrains us: the probe was validated on classification and bitext matching, not retrieval. Its authors also note that it has high precision but low recall, meaning passing the threshold does not guarantee competence. We therefore use it as a language-level feature and a hard gate that drops the dense score, and we test rather than assume that its threshold transfers to retrieval.

Concretely, for query $q$ in language $L$ we score documents with $s(q,d) = \alpha(q,L)\, \tilde s_{\text{BM25}}(q,d) + (1-\alpha(q,L))\, \tilde s_{\text{dense}}(q,d)$. Here $\tilde s$ are per-query min-max normalized scores. The weight is

$$\alpha(q,L) = \sigma\big(w^\top f(q,L)\big),$$

where $\sigma$ is the logistic function and $f(q,L)$ collects label-free features. Query-level features are the fraction of highly fragmented words, mean pieces per word, query length, and the share of digits or mixed-script tokens. Language-level features are corpus fertility, the local-sensitivity score, and pretraining corpus size. The weights $w$ are fit on training languages to maximize nDCG@10 directly, by grid search over per-query oracle weights followed by regression. The model deliberately has few parameters so that it can transfer across languages. We also fit a variant restricted to language-level features, to separate how much comes from per-language versus per-query adaptation.

Baselines

The naive reference points are BM25 alone and dense alone, run on the same corpora and queries. On Mr. TyDi, the reported hybrid-over-BM25 gain of 8 MRR@100 points (Mr. TyDi) is the level our system must at least reproduce.

The strongest existing method is the fixed linear hybrid of Making a MIRACL, with its reported average of 0.578 nDCG@10 and 0.889 Recall@100. It is the strongest competitor because it is what current multilingual systems actually deploy, and its single weight is already tuned on judged data. To keep the comparison fair, we refit its single global weight using only the training languages of each leave-one-language-out split. Both methods get identical first-stage runs, the same normalization and the same grid resolution. Beating it means a higher macro-average nDCG@10 on held-out languages, with a paired randomization test across queries, and no held-out language degrading by more than one nDCG point.

Reciprocal rank fusion, the parameter-free rank combination used by Out-of-Domain Semantics to the Rescue!, is the second strong competitor because it needs no weight at all. For MIRACL and Mr. TyDi we found no reported result on Lacuna for it, so we run it ourselves on the same runs.

Two further reference points bound the result. The oracle per-language weight, tuned on the held-out language's own judgments, is the upper bound for any language-level method. The per-query oracle is the ceiling. We report the fraction of the gap between global weight and per-language oracle that each method closes. A label-free predictor built only on score-distribution features of the two retrieved lists isolates our key ingredient. It adapts per query without any vocabulary information, so if it matches our system, the vocabulary-coverage story is wrong.

Why not just…?

Why not just use reciprocal rank fusion and skip weighting altogether? This is the real threat. If rank fusion comes within noise of the per-language oracle on MIRACL and Mr. TyDi, there is little variation left to predict, and the project reduces to a negative result. The Mr. TyDi Swahili numbers argue against rank fusion being enough. When one retriever is near-useless, equal-rank fusion still injects its top documents, whereas a learned weight near 1 can suppress them. The oracle-gap measurement decides this directly, and we will report it first.

Why not just tune one weight on high-resource languages and ship it everywhere? That is exactly the global-weight baseline. The BM25 spread of 0.183 to 0.551 in Making a MIRACL suggests the best balance moves with the language. But a spread in BM25 quality does not by itself prove that the best weight moves, so the per-language oracle weights we compute will show whether a global weight is actually costly.

Why not just use a stronger dense encoder, so fusion matters less? Arctic-Embed 2.0 trains dense multilingual retrievers, fine-tuned partly on MIRACL, that perform strongly without lexical fallback. It also reports "no clear formula" for transfer to unseen languages and observes transfer degrading for them during contrastive pretraining. We found no reported MIRACL number for it on Lacuna. We therefore run the whole study with both mDPR and a strong modern encoder such as BGE-M3. We expect average adaptive-weighting gains to shrink with the stronger encoder but to persist in the high-fragmentation query bucket. If they vanish there too, the idea matters only for weak encoders, and we will say so.

Why not just repair the fragmented words with imputed embeddings, as LOVE does? Imputation changes the dense encoder's input and requires multilingual training that LOVE never attempted. Routing needs no retraining and composes with any encoder. The two are complementary, and a small side experiment applying English-trained imputation to Latin-script Mr. TyDi queries indicates whether repair makes routing redundant.

Proposed Experiments

The main experiment is leave-one-language-out on MIRACL (18 languages) and Mr. TyDi (11 languages). For each held-out language we fit the predictor on the others, predict weights from its queries and raw corpus alone, and score nDCG@10, Recall@100 and MRR@100 against all baselines. MIRACL's two surprise languages, whose judgments were withheld from development, serve as a final untouched test. Cross-benchmark transfer (fit on MIRACL, test on Mr. TyDi languages absent from training) checks that we are not fitting benchmark quirks.

The first decisive check is that the adaptive predictor closes at least half of the global-to-oracle gap on held-out languages and beats rank fusion on macro nDCG@10. The second is a bucket analysis. Sorting held-out queries by fragmentation fraction, the predicted lexical weight and the realized gain over the global weight should both rise with fragmentation. A flat curve falsifies the mechanism even if averages improve. The third tests the gate. We apply the local-sensitivity probe to each language with each encoder and check whether languages below the 0.99 threshold are exactly those whose oracle weight is near pure BM25. To obtain failing cases on judged data, we pair benchmark languages with an encoder whose pretraining excluded them, if such pairs exist in these benchmarks. Otherwise we report the probe only as a continuous feature, and state that the gate is untested on retrieval.