Removing the Language Subspace at Query Time in Zero-Shot Late-Interaction Retrieval

Multi-vector retrievers such as ColBERT keep one vector per token and score a query against a document by matching each query token to its most similar document token. When such a retriever is trained only on English and then used on other languages, each token vector mixes two things. One is what the token means. The other is which language it came from. When the query and the document are in different languages, or when one ranked list pools documents from several languages, that language component adds a similarity bonus to same-language token pairs. It also changes which document token each query token picks as its best match. We propose to estimate this language component from the document index itself, using no parallel text and no translated training data, and to remove it with a linear projection applied only to the query. A simple algebraic fact makes the correction free. Because the projection is symmetric, projecting the query gives exactly the same scores as projecting every stored document vector. The offline index stays untouched, and the correction can be added to any deployed English-trained late-interaction model, including new languages it has never seen.

The claim to test is that a large share of the zero-shot cross-lingual failure of English-trained late interaction comes from a low-rank, language-identifying subspace in token space. If so, a query-time projection closes much of the gap to heavier fixes that need parallel data or translation. If the claim is wrong, the experiments will show where the failure really lies. The two likely culprits are poor tokenization of low-resource scripts and genuine semantic misalignment, which a linear projection cannot repair.

Why this question, and what already exists

Two ingredients are already available. The late-interaction side shows that English-only training can transfer. ColBERT-XM trains shared layers on English MS MARCO triples while keeping language-specific adapters frozen, and it transfers to monolingual retrieval in other languages. It does not evaluate queries in one language against documents in another, and its token vectors still carry language information. Multilingual ColBERT-X identifies score calibration across languages as the central difficulty when one query ranks documents from several languages together. It solves this with mixed-language batches built from machine-translated MS MARCO, so it is not zero-shot. Jina-ColBERT-v2 and BGE M3 reach cross-lingual late interaction through hundreds of millions to billions of multilingual training pairs. These papers motivate the problem, but none offers a remedy that works from English supervision alone.

The disentanglement side supplies the method. Disentangled Contrastive Learning splits a single-vector embedding into a language-agnostic semantic part and a linguistic part that absorbs language identity, and it reports gains on MIRACL and mMARCO. That work covers single vectors only, relies on word-aligned parallel text from WikiMatrix, and is sensitive to how its losses are weighted. The cross-lingual alignment survey argues that zero-shot transfer is better explained by separating language-neutral from language-specific subspaces than by global geometric alignment. It also notes that nobody has applied this view to multi-vector late interaction. Token-level alignment work shows that contextual token vectors across languages align poorly at the level of word sense (about 34–35% Precision@5 on its token-level sense retrieval benchmark). It also shows that simple linear maps estimated from averaged token vectors are surprisingly stable. That finding supports trying a linear, averaging-based correction before anything heavier.

Our proposal differs from all of these in three ways. It acts at the token level inside the MaxSim score (the sum, over query tokens, of each token's best dot product with any document token). It needs no parallel or translated data. It runs entirely at interaction time, which is possible only because the representation and interaction steps are decoupled.

Core Claim and Mechanism

Let $q_i$ be the $i$-th query token vector, $d_j$ the $j$-th stored document token vector, and $U$ a matrix whose $k$ orthonormal columns span the language-identifying subspace. The corrected score of document $D$ is

$$S(Q,D) = \sum_i \max_j \, (P q_i)^\top d_j, \qquad P = I - U U^\top .$$

Because $P$ is symmetric and satisfies $P^2 = P$, we have $(Pq_i)^\top d_j = (Pq_i)^\top (P d_j)$. The score is therefore exactly what we would get by re-indexing every document with projected vectors, including which token wins each max, at the cost of one matrix-vector product per query token.

We estimate $U$ from the index without labels beyond document language. We take token vectors grouped by language, compute the between-language scatter of the per-language means, whiten it by the average within-language covariance (linear discriminant analysis), and keep the top $k$ directions. A new language needs only its own indexed documents.

A second, even cheaper correction handles calibration across languages. Subtracting the language mean $\mu_L$ from every document token changes each query-token score by the constant $q_i^\top \mu_L$. That constant is computable at query time per language and affects only pooled multilingual rankings.

Because residual-compressed indexes such as PLAID score decompressed vectors with dot products, the projection composes with compression without modification.

Research Program

The first stage is diagnostic. It measures how much language information sits in the token vectors of English-trained late-interaction models: ColBERT-XM, and a plain XLM-RoBERTa ColBERT trained on English MS MARCO. We ask how linearly separable language is as a function of $k$. We also ask how often the best-matching document token changes after projection, broken down by token type (content words, function words, subword fragments, named entities).

The second stage applies the projection at query time and sweeps $k$. It compares the full linear-discriminant subspace with mean-only centering, and with projecting only a subset of query tokens.

The third stage asks whether training can do better than post-hoc correction without breaking the English-only constraint. We fine-tune on English triples with the fixed projection inserted into the score, so the model learns to route relevance signal away from the removed directions.

Throughout, we track one known risk. MILCO found that keeping native-language tokens matters for rare named entities, and those tokens may live partly in the directions we remove. We therefore report results separately on entity-bearing queries.

Baselines

The naive reference points are the uncorrected English-trained late-interaction model and BM25. For BM25 on MIRACL, the benchmark paper Making a MIRACL reports per-language figures such as 0.183 nDCG@10 in French and 0.551 in Finnish, and its BM25 plus mDPR hybrid averages 0.578 nDCG@10. We rerun both on our splits, since BM25 is the bar that zero-shot neural models famously fail to clear in low-resource languages. Mr. TyDi reports 0.073 versus 0.396 MRR@100 for zero-shot mDPR versus BM25 in Swahili.

The strongest existing zero-shot competitor is ColBERT-XM itself. We run it with its released weights and adapters on identical collections and query sets. For its mMARCO and Mr. TyDi figures, no reported result was found on Lacuna, so our own run serves as the reference. Beating it means a consistent gain in the cross-lingual and pooled multilingual settings with no loss on monolingual retrieval. We judge significance with paired tests across queries.

The strongest non-zero-shot competitor is document translation followed by English retrieval. On CIRAL Somali, Extending Translate-Train for ColBERT-X reports 0.5118 nDCG@20 for document translation versus 0.3366 for the best end-to-end ColBERT-X. We do not expect to beat translation outright. The goal is to close a meaningful fraction of that gap with zero translation cost. We report the fraction closed and the indexing-time cost, following the accounting in Multilingual ColBERT-X.

As the heavier disentanglement competitor, we implement a token-level version of the DCL objective on the same backbone. It uses WikiMatrix alignments for the 13 languages the DCL paper used and is given the same tuning budget of grid points over loss weights as our $k$ sweep. It uses parallel data we deliberately forgo, so matching it would be the practical win.

Three ablations isolate the key ingredient. A random rank-$k$ projection controls for the loss of dimensions alone. Mean-only centering separates calibration effects from argmax effects. Projecting both query and index is a numerical check that it equals query-only projection.

Why not just…?

Why not just translate the documents into English? On low-resource languages this is currently the best option. The CIRAL result above shows a large margin in Somali, and we treat it as the ceiling rather than something to argue away. Its cost grows with collection size and it must be redone for every language added. Multilingual ColBERT-X found native indexing reached 98% of translation MAP on CLEF with 84% less indexing time. The question our experiments settle is how much of the remaining low-resource gap a free, query-time correction recovers.

Why not just use a massively pretrained multilingual retriever like BGE M3? Where it covers a language well, it will likely win. BGE M3 reports 71.5 average nDCG@10 on MIRACL, but it was trained on 1.2 billion pairs, which is a different regime from English-only supervision. Our correction can also be applied on top of such models. If it still reduces same-language preference in pooled rankings there, it is complementary rather than competing.

Why not just apply DCL-style disentanglement during training? This is the real threat. If a trained token-level disentangler with parallel data clearly beats the query-time projection, the linear-subspace story is incomplete. The matched-budget DCL baseline tests this directly. We also report the reverse finding if it occurs: parity would mean parallel data buys little once the language subspace is removed.

Why not just subtract a per-language mean vector? This is the simplest possible version, and it is included as an ablation. Mean-centering alone changes only scores across languages, never which document token wins each max. If it matches the full projection, the paper's contribution reduces to a calibration fix, and we will say so.

Proposed Experiments

We evaluate true cross-lingual retrieval on CIRAL (English queries, Hausa, Somali, Swahili and Yoruba news), reporting nDCG@20. We evaluate pooled multilingual ranking on CLEF 2001–2003, reporting MAP and P@10, and cross-lingual open-domain retrieval on MKQA, reporting Recall@100. MILCO reports 76.6 zero-shot Recall@100 there, though with 594M parallel sentences. Monolingual non-regression is checked on MIRACL (nDCG@10, Recall@100) and Mr. TyDi (MRR@100).

The decisive diagnostic uses mMARCO. Because its passages are translations of one another, we can pool each relevant passage in every language and measure how often the query-language copy outranks semantically identical copies in other languages. With language bias removed, this "same-language preference rate" should drop toward chance while nDCG@10 is maintained.

We make two falsifiable predictions. First, gains should be largest where the script is well covered but the language differs from the query's. Second, gains should be near zero for Yoruba under base XLM-R, where the extension paper traced failure to missing pretraining coverage rather than language bias. If gains instead track tokenization quality, the projection is fixing the wrong thing. If the random projection matches ours, the subspace carries no special language signal.