Overcoming the Scalability Bottleneck of Cross-Encoder Ranking

65 concepts59 papers1 proposal

In modern information retrieval and semantic matching, researchers face a fundamental tension between accuracy and speed. On one hand, Cross-Encoders (or "joint encoders") offer state-of-the-art precision by feeding a query and a candidate document into a transformer simultaneously. This allows for token-level cross-attention, where every word in the query can interact with every word in the document. On the other hand, this deep interaction creates a massive computational bottleneck that makes Cross-Encoders nearly impossible to use for searching large-scale databases in real-time.

The Quadratic Complexity Problem

The primary reason Cross-Encoders struggle with latency is their lack of modularity. In a standard Bi-Encoder (or "dual-encoder") setup, document representations can be pre-computed and stored in a vector index. When a user submits a query, the system only needs to encode the query once and perform a fast dot-product search against the pre-computed index.

As Razin et al. and Zaharia and Khattab explain, Cross-Encoders break this paradigm. Because the query and document are entangled from the very first layer of the neural network, the model must perform a full forward pass for every unique query-document pair. If a system needs to rank 1,000 candidates for a single query, it must run the transformer 1,000 times. As Matej Kocián et al. note, this scaling behavior is a "recipe for a multi-second lag," which is unacceptable for web search or interactive dialog systems.

Strategy 1: The Multi-Stage Pipeline

To manage this overhead, the most common industry solution is a two-stage "retrieve-and-rerank" architecture. A fast, "coarse" retriever (like BM25 or a Bi-Encoder) first narrows down millions of documents to a small pool of candidates (e.g., the top 100). Only then is the expensive Cross-Encoder used to rerank this small subset.

Hu et al. demonstrate this with TOSS, which achieves high accuracy while running 1,000 times faster than a standalone high-precision model. Similarly, Mao et al. proposed RIDER, a training-free reranker that uses the reader's own predictions to bypass some of the supervised reranking costs. However, even in these pipelines, the Cross-Encoder remains the primary latency bottleneck, especially as the number of labels or candidates grows (Li et al.).

Strategy 2: Distilling Interaction into Representations

A more ambitious direction involves "distilling" the power of a Cross-Encoder into a faster Bi-Encoder. The goal is to train a student model that retains the speed of independent encoding but mimics the sophisticated similarity scores of a Cross-Encoder teacher.

  • Knowledge Distillation: Razin et al. showed that by distilling BERT into a Siamese student, they could achieve a 13,500x speedup with only a minor dip in accuracy.
  • Joint Training: Ren et al. introduced RocketQAv2, which uses dynamic listwise distillation to allow the retriever and reranker to learn from each other simultaneously, improving the quality of the efficient first-stage retriever.
  • Dual-View Learning: Xingyi Cheng proposed DvBERT, which injects fine-grained word-level interaction knowledge directly into sentence embeddings, bridging the gap between slow interaction models and fast representation models.

Strategy 3: Late Interaction and Decoupling

Rather than choosing between "early fusion" (Cross-Encoder) and "no fusion" (Bi-Encoder), some researchers propose a middle ground: Late Interaction.

The most prominent example is ColBERT, introduced by Zaharia and Khattab. ColBERT encodes queries and documents independently (allowing for pre-computation) but employs a "MaxSim" operation at the very end to model fine-grained interactions. This achieves a 170x speedup over standard BERT rankers. Similarly, Nie et al. developed DC-BERT, which decouples the encoding process and uses a lightweight interaction layer to retain 98% of BERT's accuracy while being fast enough for real-time question answering.

Strategy 4: Beyond Pointwise Scoring

Finally, new architectures are moving away from "pointwise" scoring—where each pair is scored in isolation—toward "listwise" processing. Paliwal et al. introduced CROSS-JEM, which scores an entire list of items in a single transformer pass. By leveraging token redundancies across the candidate list, they achieved a 4x speedup over traditional Cross-Encoders, making high-precision ranking more viable for massive-scale production environments.

Key Foundational Work

Joint Retrieval-Extraction Training for Evidence-Aware Dialog Response Selection (2021) — contributed 2 concept elements to this direction.
Expansion via Prediction of Importance with Contextualization (2020) — contributed 1 concept elements to this direction.
DC-BERT: Decoupling Question and Document for Efficient Contextual Encoding (2020) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_cross_encoder_inference_latency_598cbee6,
  title = {cross-encoder inference latency},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/cross-encoder-inference-latency/rdr_598cbee663da4d0ab2491512e4e49a05},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_cross_encoder_inference_latency_598cbee6_v1,
  title = {cross-encoder inference latency},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/cross-encoder-inference-latency/rdr_598cbee663da4d0ab2491512e4e49a05/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake