When a language model answers questions from retrieved passages, a reranker first sorts those passages. Rerankers are usually built and judged on topical relevance: does the passage discuss what the question asks? This note argues that relevance and usefulness come apart sharply once several languages are involved. A Swahili passage and an English passage can state the same fact and be equally relevant, yet the model reading them may answer correctly from one and fail on the other because it reads Swahili poorly. We propose to rerank passages by how much the reader can actually use them, not by how relevant they look. The score comes from the reader's own attention in a single forward pass. We choose which attention heads to trust using English data only, then apply them unchanged to other languages.
The practical stakes are cost and access. Today the strongest cross-lingual rerankers either call a frontier model on long listwise prompts or translate every candidate passage into English first. Both are expensive, and neither asks whether the reader benefits. If a cheap, reader-internal signal tracks usefulness across languages without any target-language supervision, multilingual retrieval-augmented question answering could get better answers and lower cost from the same component.
Core Claim
In mixed-language candidate pools, the passage ranking that maximizes answer quality differs from the ranking that maximizes relevance. The difference is systematic: it follows how well the reader handles each language. We further claim that attention heads selected in English because their attention predicts a passage's usefulness to the reader will carry that property to other languages without retraining. Heads selected for relevance will not. This yields a single-pass reranker that improves end-to-end answer F1 over relevance-based rerankers on parallel multilingual pools, at a small fraction of their cost.
We define usefulness concretely. For question $q$, candidate passage $d$, reader model $M$, and gold answer $a$, the utility of $d$ is the gain in answer log-likelihood from adding it:
$$u(d) = \log p_M(a \mid q, d) - \log p_M(a \mid q).$$
For each attention head $h$, let $s_h(d)$ be the attention mass that the question tokens place on $d$'s tokens when all candidates are packed into one prompt. We select heads by how well $s_h$ ranks candidates by $u$ on English data (rank correlation within each query's pool). The final score for each candidate is the average of $s_h(d)$ over the selected heads. At test time the reranker needs no gold answer and no per-candidate generation.
Where This Sits in Prior Work
Three papers establish the gap between relevance and usefulness, all in English. D.Va scores each candidate demonstration by the reader's loss on a proxy "anchor" example and shows that usefulness depends on the model. GENICL trains a retriever on the language model's own likelihood feedback and beats E5 by more than 5 points on PAWS, BoolQ and SNLI. BAR trains a tool retriever against near-duplicate queries that require opposite behavior. These papers motivate our utility target. They differ from our proposal in two ways: they either run a forward pass per candidate (D.Va) or train a separate retriever (GENICL, BAR), and none tests another language.
Our scoring method comes from Query-Focused Retrieval Heads. That paper finds heads whose question-to-context attention identifies relevant passages and reranks with them in one forward pass, beating RankGPT on BEIR. It selects heads by attention to gold relevant context, evaluates only English, and explicitly leaves cross-lingual robustness open. We keep its single-pass mechanism but change two things: the criterion for choosing heads (utility instead of relevance) and the evaluation regime (languages the heads were never selected on).
Two papers constrain the design. Adeyemi et al. evaluate listwise LLM reranking on CIRAL, a benchmark pairing English queries with passages in four African languages. Translating passages into English beforehand gains up to 7 nDCG@20 points, which tells us that the reader's language ability matters a great deal. They also find that GPT-4's own query translations beat Google Translate (0.3835 vs 0.3602 average nDCG@20), which suggests that signals internal to the model can beat external pipelines. Neither result tells us which passages help downstream answers. Disentangled Contrastive Learning shows that multilingual encoders mix meaning with language-specific patterns. That finding is a warning for us: head attention could be similarly contaminated by language, and our cross-lingual checks target exactly this failure.
For cost, ASRank amortizes one LLM call per query and hands scoring to a small T5, cutting cost from about \$700 to \$15 relative to RankGPT, but only in English. Zhuang et al. show that base models can score passages by query likelihood (51.1 nDCG@10 on a BEIR subset) and that general instruction tuning hurts this (45.1). That instruction-tuning penalty is one reason we score with attention rather than with generated likelihoods.
Research Program
The first step builds a test in which relevance and usefulness can diverge in a controlled way. MLQA is multi-way parallel: the same Wikipedia paragraph and answer exist in up to seven languages. Its question and context languages can also be set independently. For each question, in English, Arabic, Hindi or another MLQA language, we build a pool of the gold paragraph in several languages plus distractor paragraphs. The distractors come from a BM25 plus mDPR first stage over the same Wikipedias. All parallel gold copies are equally relevant by construction. Any difference in the reader's answer quality between them is therefore a difference in usefulness alone.
The reader is an open multilingual instruction model in the 8B range, for example Llama-3.1-8B, the model D.Va evaluates. It is asked to answer in the question's language, and we score it by F1 against MLQA's question-language answer. For every pool we compute the exact utility $u(d)$ of each candidate by brute force. This gives us an oracle ranking and a measurement of how large the relevance–usefulness gap really is. MLQA's reported transfer gap motivates expecting a large effect: XLM reaches 68.0 F1 on Spanish, 54.8 on Arabic and 48.8 on Hindi, with roughly a 40% drop in exact match relative to English.
The second step selects heads. Using English-question, English-context pools only, we compute each head's rank correlation between $s_h$ and $u$ and keep the top heads. As a control, we also select a relevance-head set with the Query-Focused Retrieval Heads criterion, using the same data and the same reader.
The third step tests transfer. We freeze both head sets and apply them to the non-English and mixed-language pools. We then measure how well each set's ranking correlates with the oracle utility ranking, and what top-$k$ answer F1 each achieves.
The fourth step checks that utility heads have not simply stopped detecting relevance. We rerank on MIRACL and CIRAL, where only relevance labels exist, and report nDCG. The expected pattern is that nDCG stays near that of relevance heads while answer F1 rises on MLQA. If nDCG collapses, the utility heads are exploiting something other than content, and we would report that as a failure.
Baselines
The naive reference is the first-stage ranking with no reranker, a BM25 plus mDPR hybrid. On MIRACL this hybrid reports an average nDCG@10 of 0.578 and Recall@100 of 0.889 (MIRACL). The same pools and top-$k$ go to the reader in every condition, so the only thing that differs across systems is the ordering.
The strongest existing method on cross-lingual reranking quality is listwise GPT-4 reranking as run by Adeyemi et al.. On CIRAL it reports 0.3835 average nDCG@20 with self-translated queries. We run RankGPT-style listwise reranking with the same candidate pools and window settings on CIRAL, MIRACL and our MLQA pools. We report answer F1 and cost in tokens and dollars. Beating it on CIRAL nDCG@20 is not our goal. The bar is to match or exceed its downstream MLQA F1 at a small fraction of its per-query cost, while staying within a few nDCG points on CIRAL.
The strongest single-pass competitor is the relevance-head reranker from Query-Focused Retrieval Heads, run on the identical reader, prompt packing and head budget. The file reports no result for it on MLQA, MIRACL or CIRAL, so its multilingual numbers will be produced here. This comparison doubles as the key ablation, because the two systems differ only in how heads are chosen. Utility heads must beat relevance heads on MLQA answer F1 in non-English and mixed pools to support the core claim.
We also include pointwise query-likelihood reranking in the style of UPR, using the same reader, as the generative-scoring reference. We include an ASRank-style scent reranker with a multilingual T5 scorer as the amortized-cost reference. Finally, a D.Va-style per-candidate utility estimate, which requires $k$ reader passes, serves as an expensive upper bound on what utility scoring can achieve.
A second ablation selects heads on target-language data instead of English. The gap between English-selected and target-selected heads measures directly how much is lost by transferring zero-shot.
Why not just…?
Why not just translate everything into English and rerank there? The CIRAL results show that passage translation gains up to 7 nDCG@20 points. It is a real competitor, and we run it with the same reader. It pays a translation cost on every candidate before any scoring, and higher-quality external translation did not produce better rankings (0.3602 vs 0.3835 with self-translation, Adeyemi et al.). Translation also does not choose among parallel copies. Our experiment reports translate-then-rank F1 and cost side by side with utility heads. If translation wins on F1 at acceptable cost for a language, we will say so, and utility heads can instead decide which few passages are worth translating.
Why not just add a per-language bias that prefers passages in languages the reader handles well? This is the most serious threat, because much of the usefulness gap on parallel pools might be a fixed language preference. We fit this baseline explicitly: relevance-head scores plus one learned offset per language, tuned on target-language development data. That gives it supervision our method does not use. Utility heads earn their place only if they beat this baseline on MLQA F1. The most plausible way they could is by capturing variation within a language, such as passages that are relevant but whose phrasing the reader cannot use, which a per-language constant cannot express. We will report the within-language share of utility variance from the oracle so that readers can judge how much room exists.
Why not just compute utility directly per candidate, as D.Va does? It measures the right quantity, and we include it as the upper bound. Its cost grows with the number of candidates in reader forward passes before answering, and it needs an anchor example with a known answer. The question is how much of its F1 our single pass recovers. Recovering most of it at roughly $1/k$ of its reader passes is the efficiency claim. Recovering little of it would falsify the claim.
Why not just use a multilingual supervised cross-encoder? Annotated ranking data for the CIRAL languages is essentially absent (Disentangled Contrastive Learning and Adeyemi et al. both stress this). Moreover, a cross-encoder trained on relevance labels optimizes the quantity we argue is misaligned with usefulness. We run an mMARCO-trained multilingual cross-encoder on the MLQA pools as one additional reference.
Proposed Experiments and Decisive Checks
The main table reports MLQA answer F1 at top-1 and top-3 across question and context language pairs and mixed pools. It covers every system above, alongside CIRAL nDCG@20 and MIRACL nDCG@10. Cost is reported as reader forward passes, estimated FLOPs following Efficiency-Effectiveness Reranking FLOPs, and wall-clock time per query.
Three outcomes decide the proposal. The first check is whether the oracle shows a meaningful relevance–usefulness gap on parallel pools at all. If the reader answers about equally well from every language copy, the premise fails, and we report that as a finding about modern multilingual readers. The second check is whether English-selected utility heads beat both relevance heads and the per-language-bias baseline on non-English F1. If they do not, the utility criterion adds nothing beyond a language prior. The third check is whether utility heads keep nDCG on MIRACL and CIRAL close to relevance heads. If nDCG collapses, they are tracking something other than content.
A supporting analysis plots, per language, how strongly head attention correlates with utility against MLQA transfer difficulty. This asks whether transfer weakens for languages with heavier interference between language and meaning, the effect Disentangled Contrastive Learning identified in encoders.
References
- D.Va: Validate Your Demonstration First Before You Use It
- Learning to Select In-Context Demonstration Preferred by Large Language Model
- Beyond Semantic Similarity: Reducing Unnecessary API Calls via Behavior-Aligned Retriever
- Query-Focused Retrieval Heads Improve Long-Context Reasoning and Re-ranking
- Zero-Shot Cross-Lingual Reranking with Large Language Models for Low-Resource Languages
- Disentangled Contrastive Learning for Zero-Shot Multilingual Dense Retrieval
- ASRank: Zero-Shot Re-Ranking with Answer Scent for Document Retrieval
- Open-source Large Language Models are Strong Zero-shot Query Likelihood Models for Document Ranking
- MLQA: Evaluating Cross-lingual Extractive Question Answering
- Making a MIRACL: Multilingual Information Retrieval Across a Continuum of Languages
- Improving Passage Retrieval with Zero-Shot Question Generation
- Efficiency-Effectiveness Reranking FLOPs for LLM-based Rerankers