Search systems usually decide what language a query is written in before they search, then send it to that language's index. For a two-word query in Danish, Swedish or Norwegian, that decision is often a coin flip. Sometimes it is not even well defined, because the same string is valid in several of these languages. The decision also gets worse over time. Language identifiers are trained on old text and have never seen this year's brand names, politicians or slang. LLM rerankers have the same problem: their knowledge stops at a training cutoff. We propose to stop forcing the language choice early. The query is searched under every plausible language hypothesis. The resulting ranked lists are merged with reciprocal rank fusion, a merging rule that uses only rank positions and not raw scores. Each list's weight comes from the language identifier's confidence, and that confidence is deliberately softened when the query contains strings the identifier has probably never seen. The merged pool is then reranked by an LLM that scores how likely each passage is to "generate" the query. Our hypothesis is that this scoring direction relies less on the model's stale internal knowledge than asking it directly whether a passage is relevant.
The opportunity is that two failures usually studied apart, short-text language ambiguity and knowledge obsolescence, meet on the same queries: short ones that name something new. Neither failure is measured on that slice today. We will build a time-split benchmark that isolates it and test whether deferring the language decision recovers the documents that early routing loses.
Core Claim
For short queries among closely related languages, a single early language label hides a retrieval failure. That failure grows with how new the query's vocabulary is relative to the training data of the language identifier and the reranker. Two components should fix most of it. The first is fusion over per-language retrieval runs, weighted by the identifier's language probabilities after a novelty-dependent softening. The second is query-likelihood reranking. The claim stands or falls on one ablation: posterior-weighted, novelty-softened fusion against plain equal-weight fusion of the same runs, measured separately on queries about pre-cutoff and post-cutoff entities.
The fused score of document $d$ is
$$S(d) = \sum_{\ell} p(\ell \mid q)^{\tau(q)} \frac{1}{k + r_\ell(d)} + w_0 \frac{1}{k + r_0(d)}.$$
Here $p(\ell \mid q)$ is the multi-label identifier's probability that query $q$ is valid in language $\ell$, and $r_\ell(d)$ is $d$'s rank in the run on language $\ell$'s index. The term $r_0$ is the rank in a language-agnostic multilingual dense run with fixed weight $w_0$, and $k=60$ is the standard smoothing constant. The exponent $\tau(q) \in [0,1]$ shrinks toward 0 as the query's novelty rises, which flattens the weights toward uniform. We measure novelty as the fraction of the query's character n-grams and tokens that are absent from the identifier's training corpus.
Grounding in Prior Work
Several papers motivate the gap. SLIDE builds multi-label identifiers for Danish, Swedish, Bokmål and Nynorsk. It shows that 16% of sentences under six words are valid in more than one of these languages, and it explicitly leaves open how multi-label outputs should be routed into retrieval. We take up exactly that open question. Language Identification of Search Engine Queries classifies two-to-three-word queries with a decision tree. It reaches 85.6% from linguistic features alone and 94.5% with user location, but it is single-label. Language Identification: The Long and the Short of the Matter explains why short texts fail: there are too few character n-grams for the counts to resemble a language profile. Together these establish that early hard routing is fragile. None of them measures the downstream retrieval cost, or how that cost changes as vocabulary drifts.
On the obsolescence side, RealTime QA runs a weekly news benchmark and attributes 34% of system errors to retrieval failures, against 13% to reading comprehension. That puts the retrieval stage, not the reader, at the center of staleness. Mitigating Temporal Misalignment by Discarding Outdated Facts predicts fact lifespans so the system knows when to trigger retrieval. We borrow its logic of discounting confidence according to expected staleness. The difference is that we apply it to a language identifier's confidence rather than to an answer's.
Some prior work supplies the method. UPR reranks by $P(\text{query} \mid \text{passage})$ under a frozen language model and reports 6–18% absolute top-20 gains over unsupervised retrievers. We reuse it. Our new hypothesis is that it is more robust than relevance-judgment prompts on new entities, because a new name the model has never seen can still be copied from the passage into the query. MMMORRF introduces weighted rank fusion with heuristic "news-likeness" weights and reports a 4.2% gain over plain fusion. It flags fixed heuristic weights as an open problem; we replace them with calibrated, novelty-aware posteriors.
One paper constrains the design. Cross-Dialect Information Retrieval finds that a zero-shot Llama-3-8B listwise reranker scores 0.723 nDCG@10 on dialectal German, below plain BM25 at 0.737. Prompted rerankers clearly break on unfamiliar vocabulary. A design that trusts the reranker unconditionally is therefore ruled out. The final ranking will fuse the reranker's list with the first-stage lists rather than replace them. The TREC 2023 NeuCLIR overview adds that merging across languages tends to favor the languages where models are more confident. We will check whether posterior weighting reproduces that unfairness.
Research Program
We first build a time-split testbed for short Scandinavian queries. The corpus is Danish, Swedish, Bokmål and Nynorsk Wikipedia, plus news text in each language. Queries of two to four words are derived from article titles and lead phrases. A sample is annotated by native speakers with SLIDE-style multi-label tags and graded relevance; any language edition covering the intended topic can be relevant. Every query carries a novelty date: when its key entity or term first appeared in the corpus. This date lets us bucket queries relative to two cutoffs, that of the identifier's training data and that of the LLM. The point of the benchmark is to cross two variables, language ambiguity and novelty, so their interaction can be read directly.
Next we assemble the retrieval channels. These are four per-language BM25 indices with language-appropriate tokenization, plus one shared multilingual dense index. We use lexical runs because dense models are known to miss exact new identifiers. Language probabilities come from SLIDE-fast, which is cheap at 0.16 ms per query on CPU. Its probabilities are temperature-calibrated on held-out pre-cutoff queries. The softening schedule $\tau(q)$ is fit on a pre-cutoff development split only, so post-cutoff evaluation stays clean.
Finally we compare two families of rerankers on the fused top-100. The first is query-likelihood scoring in the style of UPR. The second is a listwise relevance prompt in the style of RankGPT, using the same base LLM. To separate the reranker effect from Scandinavian ambiguity, we repeat the reranker comparison in English on RealTime QA retrieval. There we split questions by whether the answer entity postdates the model's cutoff.
Baselines
The naive reference point is hard routing. Take the top language label from SLIDE-fast, search only that index with BM25, and do not rerank. This is what most deployed systems do. Our method must beat it on post-cutoff, multi-label queries by a margin larger than the run-to-run variance across query bootstrap samples. Otherwise there is no problem worth solving.
The strongest existing competitor is a language-agnostic stack: the multilingual dense index searched with no language identification at all, then reranked by the same LLM. It is strong because closely related languages share most of their vocabulary, and it avoids routing errors entirely. It gets the same candidate depth, the same reranker and the same LLM call budget. To count as beaten, our system must improve nDCG@10 on the post-cutoff bucket while not losing on the pre-cutoff bucket.
The second strong competitor is equal-weight fusion of all five runs. It is the standard zero-shot hybrid, needs no weights, and may capture most of the benefit by itself.
For anchoring, the file gives no reported retrieval result on Lacuna for short-query Scandinavian retrieval. The closest reported numbers concern components. On language identification, SLIDE-base reaches 96.4% exact-match accuracy, SLIDE-fast 93.4% and FastText 77.7%, all from SLIDE. On temporal retrieval, GPT-3 with Google Search reaches 66.5% multiple-choice accuracy on RealTime QA (RealTime QA). On prompted reranking, the best TREC 2023 prompt run scores 0.6994 nDCG@10 on MS MARCO v2 (TREC 2023 DL overview).
The key ablations isolate the two ingredients. The first compares posterior weights with $\tau$ fixed at 1, posterior weights with $\tau$ fixed at 0 (which is equal-weight fusion), and the novelty-dependent $\tau(q)$. The second swaps query-likelihood reranking for listwise relevance prompting while holding everything else fixed. All variants share indices, candidate depth and LLM budget. The only tuning is calibration and the $\tau$ schedule, done on the pre-cutoff development split.
Why not just…?
Why not just drop language identification and use one multilingual dense index? This is the most serious threat. On pre-cutoff queries it may match our system. Our reason to expect a gap after the cutoff is that dense encoders are documented to miss exact entity strings, and new names are exactly such strings. The language-agnostic baseline above settles the question directly. If it matches us on the post-cutoff bucket too, we will report that early language decisions are simply unnecessary for this language family. That is a useful negative result, and the dense run already sits inside our fusion.
Why not just fuse all runs with equal weight? Equal weights are the $\tau = 0$ endpoint of our method. Our method collapses to this when the query is maximally novel, and it may well be competitive overall. What weighting could add is precision on unambiguous, familiar queries, where flooding the pool with three wrong-language lists hurts. The $\tau$ ablation will show whether that gain exists. If it does not, the contribution reduces to fusion plus the reranker finding, and we will say so.
Why not just use a better identifier, or add user location as Carter et al. did? A better identifier still forces a single label on the 16% of short texts that are genuinely valid in several languages (SLIDE), and no identifier has seen post-cutoff names. Location raised accuracy to 94.5% in Language Identification of Search Engine Queries, but it is often unavailable, and it is weak across Scandinavian borders. Where location exists it can enter as a prior on $p(\ell \mid q)$. We will report results with and without it.
Why not just trust a strong LLM reranker to sort it out? The dialect results show that a prompted reranker can fall below BM25 on unfamiliar vocabulary (Cross-Dialect Information Retrieval). A reranker also cannot recover documents the first stage never retrieved. The reranker-only arm, run over a single hard-routed pool, measures this directly.
Proposed Experiments
The decisive check is one comparison. We plot nDCG@10 and Recall@100 against novelty bucket and query length, for hard routing, the language-agnostic stack, equal-weight fusion and our full method. The prediction is that the gap between our method and hard routing widens with novelty and with multi-label ambiguity. We also predict that the gap to equal-weight fusion is positive on familiar queries and shrinks to zero on novel ones, which is what the $\tau$ schedule implies.
The second check is reranker robustness. We compare query-likelihood and listwise relevance prompting on pre- versus post-cutoff queries, both on the Scandinavian testbed and on RealTime QA. The prediction is that listwise reranking loses more after the cutoff. If both rerankers degrade equally, the copy-based explanation is wrong.
The third check addresses fairness. Using NeuCLIR-style per-language analysis, we test whether posterior weighting systematically under-retrieves Nynorsk, the smallest of the four varieties, compared with equal-weight fusion.
Finally, we report the latency of the five-run fusion relative to hard routing. Most of the extra cost is the additional BM25 lookups, which are inexpensive.
References
- Multi-label Scandinavian Language Identification (SLIDE)
- Language Identification of Search Engine Queries
- Language Identification: The Long and the Short of the Matter
- RealTime QA: What's the Answer Right Now?
- Mitigating Temporal Misalignment by Discarding Outdated Facts
- Improving Passage Retrieval with Zero-Shot Question Generation
- MMMORRF: Multimodal Multilingual MOdularized Reciprocal Rank Fusion
- Cross-Dialect Information Retrieval: Information Access in Low-Resource and High-Variance Languages
- Overview of the TREC 2023 NeuCLIR Track
- Overview of the TREC 2023 Deep Learning Track