Does Passage Aggregation Survive the Language Switch?

When a document is too long for a neural ranker to read in one pass, the standard fix is to cut it into passages, score or encode each one, and combine the results into one document score. In English, the best way to combine them is to feed each passage's vector into a small second Transformer that can compare passages against each other. This beats simply taking the best passage's score. Separately, multilingual encoders let a ranker trained only on English relevance labels work, with no further training, on Chinese, Arabic, or Swahili; this is called zero-shot cross-lingual transfer. Nobody has checked whether the learned combiner survives that transfer. There is a specific reason to expect it might not. The combiner never sees words, only passage vectors, and multilingual passage vectors carry a strong "which language is this" direction. A combiner trained on English vectors may treat that direction, along with English habits about where relevant text sits, as part of what relevance looks like.

This note proposes a controlled test of that claim, together with a cheap repair. The claim is that the English advantage of representation aggregation over best-passage pooling shrinks or reverses in unseen languages. We expect two causes. First, the combiner's inputs are shifted by a language-specific offset. Second, it has learned an English positional prior that relevant text comes first. We expect the advantage to come back when two things are done: each language's mean passage vector is subtracted before aggregation, and the training documents are shuffled so that relevant passages appear at arbitrary positions. The note also builds the missing evaluation needed for this test: document-level multilingual ranking with native queries, reported separately by where in the document the relevant text sits.

Core Claim and What Is New

Two strands of prior work set up the question. The first is English representation aggregation. PARADE encodes each passage to a [CLS] vector and aggregates the vectors with a shallow Transformer, beating BERT-MaxP on Robust04 and GOV2. MPII at NTCIR-15 WWW-3 reports the same ordering on English web documents, with the full Transformer at 0.6897 nDCG@10 against 0.6337 for max aggregation. Both studies use English-only encoders and English-only test sets. The second strand is cross-lingual document ranking. Shi & Lin fine-tune mBERT on English tweets with frozen token embeddings and combine BM25 with the top-3 sentence scores. Shi, Bai & Lin instead filter each document down to sentences that contain query terms. Both combine sentences as independent scalar scores and never train an aggregator over vectors.

The proposal therefore differs from each strand in a specific way. It is the first test of learned vector aggregation under zero-shot language transfer. It separates two candidate reasons for failure, language offset and positional prior, and pairs each with its own cheap remedy.

The disentangled model of Huang et al. plays a supporting role. It shows that language features contaminate multilingual passage vectors, and it removes them using learned orthogonal subspaces. However, it scores isolated passages and never aggregates them. We use its finding as motivation and its model as a heavier alternative remedy, but we lead with plain mean-centering. If centering works, the fix is nearly free.

The positional hypothesis comes from Boytsov et al., who found two things. English long-document models gain at most about 5% over first-passage truncation (FirstP), because benchmark relevance sits in the first 256–512 tokens. When relevant text is moved past token 512 (the MS MARCO FarRelevant diagnostic), most models fall to near-random performance. A combiner that has learned "early is relevant" from English web and news documents faces a different situation in other languages, whose Wikipedia articles and documents may be structured differently. That combiner could fail in two ways: it could miss relevant passages that appear late, and it could over-reward passages simply because they appear early.

The repair is simple. Let $h_i$ be the vector for passage $i$ of a document written in language $\ell$. Let $\mu_\ell$ be the mean passage vector computed over unlabeled text in that language, which uses no relevance labels, so the setting stays zero-shot. The combiner receives

$$\tilde h_i = h_i - \mu_\ell,$$

and during English training the passage order is randomly permuted with probability $p$. Position embeddings are kept, but the combiner can no longer rely on them alone.

Research Program

Encoders and training. We fix a multilingual passage encoder. The main choice is multilingual-e5-large-instruct, which MMTEB reports as the best public multilingual embedder; XLM-R is a secondary encoder. All rankers are trained only on English MS MARCO document ranking, using 150-word passages with 50-word overlap as in the MPII setup. Following Shi & Lin, freezing versus unfreezing the token embeddings is a crossed factor.

Aggregators. We train the whole aggregation family under one recipe: FirstP, MaxP, top-3 averaging, attention aggregation, and the full Transformer aggregator. Each is trained with and without centering and with and without order permutation.

Diagnostics. A linear probe for language identity is run at the combiner's input and at its output. This measures how much language signal survives into the relevance decision. We also measure the combiner's attention mass by passage position, in English and in each target language.

Evaluation data. The key piece of building work is a document-level test set with native queries. MIRACL and Mr. TyDi judge Wikipedia passages written by native speakers, and every passage belongs to a parent article. We lift these labels to the article level. An article is relevant if any of its judged passages is relevant, and its graded relevance is the number of relevant passages it contains. Articles are then re-ranked in full.

Each query is tagged by the position of the first relevant passage in its article. This lets us report results in bands: first passage, passages 2–4, and later. We also build a FarRelevant-style variant by moving the relevant passage deeper into its own article.

The lifted labels have two biases, and we report both rather than hide them. First, "any passage relevant" is a max-like label, which favors MaxP by construction. The graded count label and the native document judgments of NTCIR-8 Chinese partly offset this. Second, unjudged passages are treated as unknown, not as non-relevant, and evaluation uses measures that tolerate missing judgments.

Baselines

The do-nothing reference is language-tuned BM25 over whole documents. This is the floor every neural method must clear. On Mr. TyDi, BM25 beat zero-shot dense retrieval badly on Swahili, 0.396 versus 0.073 MRR@100 (Mr. TyDi). BM25 also supplies the top-1,000 candidates that every re-ranker scores, so all systems see identical documents.

The strongest existing cross-lingual document method is Shi & Lin's 3-sentence mBERT model, which linearly combines BM25 with the top-3 sentence scores. On NTCIR-8 Chinese it reports 0.4612 AP against 0.4065 for BM25 (Cross-Lingual Relevance Transfer for Document Retrieval). We reproduce it on the same candidates. We run it both with the original mBERT and with our encoder, so that a stronger encoder is not credited to aggregation. Beating it means higher AP on NTCIR-8 Chinese, significant on paired tests, with the same encoder on both sides.

The strongest existing aggregator is the PARADE full Transformer, run as published, with raw vectors and no permutation. Its English reference number is 0.6897 nDCG@10 on NTCIR-15 WWW-3 English (MPII). There is no reported result on Lacuna for PARADE in any non-English language; producing that number is part of this work.

MaxP and FirstP use the identical encoder, training data, and hyperparameter search budget. The same number of tuning trials is used on the English development set only.

Ablations isolate the proposal's two ingredients: centering alone, permutation alone, and both together. They also include Huang et al.'s disentangled semantic projection as a heavier substitute for centering.

The proposal counts as successful if it meets three conditions. The raw Transformer aggregator's margin over MaxP in target languages must fall well below its English margin. The combined remedy must restore that margin on average across MIRACL languages. The gain must concentrate in the non-first-position bands.

Why not just…?

Why not just truncate to the first passage? On Wikipedia this is a real threat. Lead sections summarize the article, and Boytsov et al. found that FirstP is within 5% of everything on English benchmarks. We do not argue this away. The position-banded results settle it: if FirstP ties the aggregators in the first band and collapses in later bands, then aggregation matters exactly where FirstP cannot reach. If FirstP ties everywhere, we will report that multilingual Wikipedia is not a test of long-document ranking at all.

Why not just take the max passage score? MaxP has no cross-passage parameters that could absorb English habits, so it may transfer better than the full aggregator, and our hypothesis predicts exactly this for raw vectors. The lifted labels also favor it. The proposal therefore stands or falls on whether the graded labels, the NTCIR-8 native judgments, and the later position bands show the centered aggregator ahead. If they do not, the conclusion is that MaxP should be the default for zero-shot ranking. That is a useful outcome in its own right.

Why not just translate documents to English and run English PARADE? Shi, Bai & Lin found search-time document translation accurate but costly. They also found that word-by-word dictionary translation beat full machine translation for Hindi and Bengali, and mMARCO documents lexical mismatch introduced by translation. We run translate-test on a subset of languages as an upper-reference system and report its per-query cost. Centering is only interesting if it recovers a meaningful share of the translation gain at none of that cost.

Why not just use the disentangled encoder of Huang et al. and skip centering? It may work as well or better. That is why it appears in the ablations as a direct competitor rather than being cited as prior art. Centering needs no extra training. If the learned subspaces do not clearly beat it, the simpler fix wins.

Proposed Experiments

The first decisive check measures the transfer of the aggregation gap. We compute the raw Transformer aggregator's advantage over MaxP on English MS MARCO documents and on NTCIR-15 English. We then compute the same advantage on each MIRACL language and on NTCIR-8 Chinese. Our hypothesis predicts that the advantage shrinks, and shrinks most in languages that are typologically distant from English.

The second check compares the language-identity probe at the combiner's input before and after centering. It then tests whether the per-language drop in probe accuracy predicts the per-language recovery of the aggregation gap.

The third check examines the attention-by-position profile and the banded nDCG@10. It asks whether permutation training moves the combiner's attention mass toward later positions, and whether it raises scores on the FarRelevant-style variant without costing first-band accuracy.

All results are reported per language, with paired significance tests. Mr. TyDi Swahili and Telugu get separate treatment, since BM25 dominance there may make every neural re-ranker look alike.