CELI: Simple yet Effective Approach to Enhance Out-of-Domain Generalization of Cross-Encoders

Xinyu Zhang, Minghan Li, Jimmy Lin

In information retrieval, Cross-Encoders represent the state-of-the-art for ranking accuracy. By performing deep self-attention across both the query and the document simultaneously, these models capture complex semantic relationships. However, a significant limitation persists: the "information bottleneck." Standard Cross-Encoders compress the entire interaction into a single representation—typically the [CLS] token—to produce a final relevance score. This compression often leads to poor generalization when the model encounters out-of-domain data.

Researchers from the University of Waterloo propose CELI (Cross-Encoder with Late Interaction), a simple architectural modification that alleviates this bottleneck. By incorporating a late interaction layer, CELI significantly improves out-of-domain robustness without compromising in-domain performance or computational efficiency.

The Information Bottleneck Problem

Modern search pipelines often utilize a two-stage approach: a first-stage retriever identifies a broad set of candidates, which a Cross-Encoder (reranker) then sorts. While effective, standard rerankers like monoBERT rely solely on the [CLS] token for the final score. This single vector must represent the entirety of the query-document relationship. In out-of-domain scenarios, where the model's pre-trained representations might be less precise, this bottleneck causes the model to overlook fine-grained details necessary for accurate ranking.

CELI: Architecture and Methodology

CELI addresses this by augmenting the standard relevance score with a token-level interaction score. The final relevance score in CELI is the sum of two components:

  1. Standard Score ($s_m$): The traditional score derived from the [CLS] token representation.
  2. Late Interaction Score ($s_l$): Inspired by the ColBERT architecture, this score utilizes contextualized embeddings for every token in the query and document.

Specifically, CELI projects the final-layer embeddings into a lower-dimensional space and performs a MaxSim operation. For each query token, the model identifies the most similar document token and sums these maximum similarity values. This dual-path approach allows the model to maintain global context while ensuring that specific token-level matches are preserved in the final ranking.

Experimental Results

The authors evaluated CELI by training on the MS MARCO dataset and testing generalization across the BEIR benchmark, which spans 13 diverse datasets.

Table 1: Comparison of monoBERT and CELI on in-domain (MS MARCO) and out-of-domain (BEIR) benchmarks, including search latency.

As shown in the results above, CELI achieves a 5% relative improvement in averaged nDCG@10 on BEIR (0.491 vs. 0.467) compared to the monoBERT baseline. Crucially, this gain does not come at the expense of in-domain accuracy; performance on MS MARCO remained stable or slightly improved. Furthermore, the computational overhead is minimal, with search latency increasing by only 0.1 seconds.

Analysis: Why CELI Works

A key finding of the study is that CELI's benefits are most pronounced for longer, more complex queries. These queries often contain multiple constraints or specific entities that are easily lost during the [CLS] token compression.

Figure 1: Performance gain (Δ nDCG@10) of CELI over monoBERT as a function of query length on the Quora and HotpotQA datasets.

The scatter plots demonstrate a clear upward trend: as query length increases, the performance delta between CELI and the baseline grows. This confirms that the late interaction mechanism effectively captures the "long-tail" of information that the [CLS] token fails to represent.

Additionally, the researchers found that the interaction itself is more critical than the dimensionality of the token embeddings. Even when projecting token embeddings down to a single dimension, CELI retained most of its performance gains, suggesting that the MaxSim operation acts as a robust "sanity check" for the global [CLS] score.

Conclusion

CELI offers a simple, "plug-and-play" upgrade for Cross-Encoder architectures. By breaking the information bottleneck through late interaction, it provides a significant boost to out-of-domain generalization. Its efficiency and retriever-agnostic nature make it a highly practical solution for real-world search systems requiring both high accuracy and robustness across diverse topics.

Summary updated

Abstract

In text ranking, it is generally believed that the cross-encoders already gather sufficient token interaction information via the attention mechanism in the hidden layers. However, our results show that the cross-encoders can consistently benefit from additional token interaction in the similarity computation at the last layer. We introduce CELI (Cross-Encoder with Late Interaction), which incorporates a late interaction layer into the current cross-encoder models. This simple method brings 5% improvement on BEIR without compromising in-domain effectiveness or search latency. Extensive experiments show that this finding is consistent across different sizes of the cross-encoder models and the first-stage retrievers. Our findings suggest that boiling all information into the [CLS] token is a suboptimal use for cross-encoders, and advocate further studies to investigate its relevance score mechanism.
BibTeX
@inproceedings{zhang2024celi,
  author = {Xinyu Zhang and Minghan Li and Jimmy Lin},
  title = {{CELI: Simple yet Effective Approach to Enhance Out-of-Domain Generalization of Cross-Encoders}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.naacl-short.16},
}
Figure