In information retrieval, Cross-Encoders represent the state-of-the-art for ranking accuracy. By performing deep self-attention across both the query and the document simultaneously, these models capture complex semantic relationships. However, a significant limitation persists: the "information bottleneck." Standard Cross-Encoders compress the entire interaction into a single representation—typically the [CLS] token—to produce a final relevance score. This compression often leads to poor generalization when the model encounters out-of-domain data.
Researchers from the University of Waterloo propose CELI (Cross-Encoder with Late Interaction), a simple architectural modification that alleviates this bottleneck. By incorporating a late interaction layer, CELI significantly improves out-of-domain robustness without compromising in-domain performance or computational efficiency.
The Information Bottleneck Problem
Modern search pipelines often utilize a two-stage approach: a first-stage retriever identifies a broad set of candidates, which a Cross-Encoder (reranker) then sorts. While effective, standard rerankers like monoBERT rely solely on the [CLS] token for the final score. This single vector must represent the entirety of the query-document relationship. In out-of-domain scenarios, where the model's pre-trained representations might be less precise, this bottleneck causes the model to overlook fine-grained details necessary for accurate ranking.
CELI: Architecture and Methodology
CELI addresses this by augmenting the standard relevance score with a token-level interaction score. The final relevance score in CELI is the sum of two components:
- Standard Score ($s_m$): The traditional score derived from the
[CLS]token representation. - Late Interaction Score ($s_l$): Inspired by the ColBERT architecture, this score utilizes contextualized embeddings for every token in the query and document.
Specifically, CELI projects the final-layer embeddings into a lower-dimensional space and performs a MaxSim operation. For each query token, the model identifies the most similar document token and sums these maximum similarity values. This dual-path approach allows the model to maintain global context while ensuring that specific token-level matches are preserved in the final ranking.
Experimental Results
The authors evaluated CELI by training on the MS MARCO dataset and testing generalization across the BEIR benchmark, which spans 13 diverse datasets.
As shown in the results above, CELI achieves a 5% relative improvement in averaged nDCG@10 on BEIR (0.491 vs. 0.467) compared to the monoBERT baseline. Crucially, this gain does not come at the expense of in-domain accuracy; performance on MS MARCO remained stable or slightly improved. Furthermore, the computational overhead is minimal, with search latency increasing by only 0.1 seconds.
Analysis: Why CELI Works
A key finding of the study is that CELI's benefits are most pronounced for longer, more complex queries. These queries often contain multiple constraints or specific entities that are easily lost during the [CLS] token compression.
The scatter plots demonstrate a clear upward trend: as query length increases, the performance delta between CELI and the baseline grows. This confirms that the late interaction mechanism effectively captures the "long-tail" of information that the [CLS] token fails to represent.
Additionally, the researchers found that the interaction itself is more critical than the dimensionality of the token embeddings. Even when projecting token embeddings down to a single dimension, CELI retained most of its performance gains, suggesting that the MaxSim operation acts as a robust "sanity check" for the global [CLS] score.
Conclusion
CELI offers a simple, "plug-and-play" upgrade for Cross-Encoder architectures. By breaking the information bottleneck through late interaction, it provides a significant boost to out-of-domain generalization. Its efficiency and retriever-agnostic nature make it a highly practical solution for real-world search systems requiring both high accuracy and robustness across diverse topics.
Summary updated