In modern information retrieval and semantic matching, researchers face a fundamental tension between accuracy and speed. On one hand, Cross-Encoders (or "joint encoders") offer state-of-the-art precision by feeding a query and a candidate document into a transformer simultaneously. This allows for token-level cross-attention, where every word in the query can interact with every word in the document. On the other hand, this deep interaction creates a massive computational bottleneck that makes Cross-Encoders nearly impossible to use for searching large-scale databases in real-time.
The Quadratic Complexity Problem
The primary reason Cross-Encoders struggle with latency is their lack of modularity. In a standard Bi-Encoder (or "dual-encoder") setup, document representations can be pre-computed and stored in a vector index. When a user submits a query, the system only needs to encode the query once and perform a fast dot-product search against the pre-computed index.
As Razin et al. and Zaharia and Khattab explain, Cross-Encoders break this paradigm. Because the query and document are entangled from the very first layer of the neural network, the model must perform a full forward pass for every unique query-document pair. If a system needs to rank 1,000 candidates for a single query, it must run the transformer 1,000 times. As Matej Kocián et al. note, this scaling behavior is a "recipe for a multi-second lag," which is unacceptable for web search or interactive dialog systems.
Strategy 1: The Multi-Stage Pipeline
To manage this overhead, the most common industry solution is a two-stage "retrieve-and-rerank" architecture. A fast, "coarse" retriever (like BM25 or a Bi-Encoder) first narrows down millions of documents to a small pool of candidates (e.g., the top 100). Only then is the expensive Cross-Encoder used to rerank this small subset.
Hu et al. demonstrate this with TOSS, which achieves high accuracy while running 1,000 times faster than a standalone high-precision model. Similarly, Mao et al. proposed RIDER, a training-free reranker that uses the reader's own predictions to bypass some of the supervised reranking costs. However, even in these pipelines, the Cross-Encoder remains the primary latency bottleneck, especially as the number of labels or candidates grows (Li et al.).
Strategy 2: Distilling Interaction into Representations
A more ambitious direction involves "distilling" the power of a Cross-Encoder into a faster Bi-Encoder. The goal is to train a student model that retains the speed of independent encoding but mimics the sophisticated similarity scores of a Cross-Encoder teacher.
- Knowledge Distillation: Razin et al. showed that by distilling BERT into a Siamese student, they could achieve a 13,500x speedup with only a minor dip in accuracy.
- Joint Training: Ren et al. introduced RocketQAv2, which uses dynamic listwise distillation to allow the retriever and reranker to learn from each other simultaneously, improving the quality of the efficient first-stage retriever.
- Dual-View Learning: Xingyi Cheng proposed DvBERT, which injects fine-grained word-level interaction knowledge directly into sentence embeddings, bridging the gap between slow interaction models and fast representation models.
Strategy 3: Late Interaction and Decoupling
Rather than choosing between "early fusion" (Cross-Encoder) and "no fusion" (Bi-Encoder), some researchers propose a middle ground: Late Interaction.
The most prominent example is ColBERT, introduced by Zaharia and Khattab. ColBERT encodes queries and documents independently (allowing for pre-computation) but employs a "MaxSim" operation at the very end to model fine-grained interactions. This achieves a 170x speedup over standard BERT rankers. Similarly, Nie et al. developed DC-BERT, which decouples the encoding process and uses a lightweight interaction layer to retain 98% of BERT's accuracy while being fast enough for real-time question answering.
Strategy 4: Beyond Pointwise Scoring
Finally, new architectures are moving away from "pointwise" scoring—where each pair is scored in isolation—toward "listwise" processing. Paliwal et al. introduced CROSS-JEM, which scores an entire list of items in a single transformer pass. By leveraging token redundancies across the candidate list, they achieved a 4x speedup over traditional Cross-Encoders, making high-precision ranking more viable for massive-scale production environments.
Key Foundational Work
Joint Retrieval-Extraction Training for Evidence-Aware Dialog Response Selection (2021) — contributed 2 concept elements to this direction.
Expansion via Prediction of Importance with Contextualization (2020) — contributed 1 concept elements to this direction.
DC-BERT: Decoupling Question and Document for Efficient Contextual Encoding (2020) — contributed 1 concept elements to this direction.