The Robustness Gap in Neural Reranking
In modern Information Retrieval (IR), "cross-encoder" rerankers (such as monoBERT) are the gold standard for accuracy. These models process a concatenated query and document through a Transformer, using the final layer's [CLS] token representation to predict a relevance score.
Despite their high performance on in-distribution data, these models face a significant "robustness gap" when applied to new domains (out-of-distribution). This paper hypothesizes that condensing the entire relationship between a query and a document into a single [CLS] vector creates a bottleneck. While the Transformer's internal attention layers process every token, the final summary may discard granular, token-level evidence crucial for generalizing to unseen topics.
Contextualized Late Interaction
To address this bottleneck, the authors propose augmenting standard [CLS]-based rerankers with a Contextualized Late Interaction layer.
- Baseline (monoBERT): A standard cross-encoder where the relevance score is a linear transformation of the last-layer
[CLS]vector. - Late Interaction Addition: Borrowing from the ColBERT architecture, the model extracts contextualized embeddings for every token in the query and document from the final layer. It then computes a "MaxSim" score: for every query token, it identifies the document token with the highest cosine similarity and sums these maximums.
- Hybrid Scoring: The final relevance score is the sum of the standard
[CLS]score and the new Late Interaction score.
Unlike bi-encoders where embeddings are computed in isolation, these token embeddings are "contextualized" because they have already interacted through the Transformer's self-attention layers before the late interaction occurs.
Key Results and Efficiency
The proposed architectural tweak provides a significant boost to out-of-distribution (OOD) generalization without degrading in-domain performance. Evaluated on the MS MARCO (in-distribution) and BEIR (OOD) benchmarks, the hybrid model showed a consistent ~5% improvement in zero-shot performance.
As shown in the table above, the average nDCG@10 on BEIR improved from 0.467 to 0.491. Crucially, the search latency only increased by approximately 100ms (from 1.18s to 1.28s). Since reranking typically only processes the top 100–1000 candidates, this overhead is negligible for most production environments.
Analysis: Why Late Interaction Helps
The benefits of late interaction are not merely due to improved keyword matching. Experiments removing exact-match tokens showed that OOD gains persisted, suggesting the model utilizes the contextualized meaning of tokens rather than simple lexical overlap.
Furthermore, the improvement is most pronounced on longer, more complex queries. This supports the theory that the [CLS] token acts as a bottleneck; as query length increases, it becomes harder for a single vector to capture every nuance required for accurate retrieval.
The gain is also model-agnostic, providing similar lifts whether using small backbones like MiniLM or larger models like ELECTRA. Additionally, the improvements remain consistent regardless of the first-stage retriever used (e.g., BM25, SPLADE, or DPR).
Conclusion
Adding a contextualized late interaction layer is a simple yet effective method to improve the zero-shot generalization of neural rerankers. By bypassing the [CLS] bottleneck and allowing for granular token-level interactions, models can better handle the diverse topics and complex queries found in out-of-distribution domains.
Summary updated