Improving Out-of-Distribution Generalization of Neural Rerankers with Contextualized Late Interaction

Xinyu Zhang, Minghan Li, Jimmy LinarXiv 2023 · DOI · arXiv

The Robustness Gap in Neural Reranking

In modern Information Retrieval (IR), "cross-encoder" rerankers (such as monoBERT) are the gold standard for accuracy. These models process a concatenated query and document through a Transformer, using the final layer's [CLS] token representation to predict a relevance score.

Despite their high performance on in-distribution data, these models face a significant "robustness gap" when applied to new domains (out-of-distribution). This paper hypothesizes that condensing the entire relationship between a query and a document into a single [CLS] vector creates a bottleneck. While the Transformer's internal attention layers process every token, the final summary may discard granular, token-level evidence crucial for generalizing to unseen topics.

Contextualized Late Interaction

To address this bottleneck, the authors propose augmenting standard [CLS]-based rerankers with a Contextualized Late Interaction layer.

  1. Baseline (monoBERT): A standard cross-encoder where the relevance score is a linear transformation of the last-layer [CLS] vector.
  2. Late Interaction Addition: Borrowing from the ColBERT architecture, the model extracts contextualized embeddings for every token in the query and document from the final layer. It then computes a "MaxSim" score: for every query token, it identifies the document token with the highest cosine similarity and sums these maximums.
  3. Hybrid Scoring: The final relevance score is the sum of the standard [CLS] score and the new Late Interaction score.

Unlike bi-encoders where embeddings are computed in isolation, these token embeddings are "contextualized" because they have already interacted through the Transformer's self-attention layers before the late interaction occurs.

Key Results and Efficiency

The proposed architectural tweak provides a significant boost to out-of-distribution (OOD) generalization without degrading in-domain performance. Evaluated on the MS MARCO (in-distribution) and BEIR (OOD) benchmarks, the hybrid model showed a consistent ~5% improvement in zero-shot performance.

Table 1: Performance comparison on MS MARCO and BEIR benchmarks, including search latency overhead.

As shown in the table above, the average nDCG@10 on BEIR improved from 0.467 to 0.491. Crucially, the search latency only increased by approximately 100ms (from 1.18s to 1.28s). Since reranking typically only processes the top 100–1000 candidates, this overhead is negligible for most production environments.

Analysis: Why Late Interaction Helps

The benefits of late interaction are not merely due to improved keyword matching. Experiments removing exact-match tokens showed that OOD gains persisted, suggesting the model utilizes the contextualized meaning of tokens rather than simple lexical overlap.

Furthermore, the improvement is most pronounced on longer, more complex queries. This supports the theory that the [CLS] token acts as a bottleneck; as query length increases, it becomes harder for a single vector to capture every nuance required for accurate retrieval.

Figure 1: Improvement in nDCG@10 as a function of query length, showing greater gains for longer queries.

The gain is also model-agnostic, providing similar lifts whether using small backbones like MiniLM or larger models like ELECTRA. Additionally, the improvements remain consistent regardless of the first-stage retriever used (e.g., BM25, SPLADE, or DPR).

Conclusion

Adding a contextualized late interaction layer is a simple yet effective method to improve the zero-shot generalization of neural rerankers. By bypassing the [CLS] bottleneck and allowing for granular token-level interactions, models can better handle the diverse topics and complex queries found in out-of-distribution domains.

Summary updated

Abstract

Recent progress in information retrieval finds that embedding query and document representation into multi-vector yields a robust bi-encoder retriever on out-of-distribution datasets. In this paper, we explore whether late interaction, the simplest form of multi-vector, is also helpful to neural rerankers that only use the [CLS] vector to compute the similarity score. Although intuitively, the attention mechanism of rerankers at the previous layers already gathers the token-level information, we find adding late interaction still brings an extra 5% improvement in average on out-of-distribution datasets, with little increase in latency and no degradation in in-domain effectiveness. Through extensive experiments and analysis, we show that the finding is consistent across different model sizes and first-stage retrievers of diverse natures and that the improvement is more prominent on longer queries.
BibTeX
@misc{zhang2023improving,
  author = {Xinyu Zhang and Minghan Li and Jimmy Lin},
  title = {{Improving Out-of-Distribution Generalization of Neural Rerankers with Contextualized Late Interaction}},
  year = {2023},
  eprint = {2302.06589},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2302.06589},
}
Figure