Two-Stage Retrieval and Re-Ranking Pipelines

26 concepts26 papers2 proposals

When building systems to search through massive collections of data—whether web pages, legal contracts, or biomedical entities—we face a fundamental engineering trade-off: speed versus depth of comprehension.

If a user submits a query, we want a system that understands the deep, nuanced semantic interactions between the query tokens and the document tokens. In modern NLP, this deep understanding is achieved via cross-attention in Transformer models. However, running a deep Transformer over millions of candidate documents for every single search query is computationally impossible in real-time.

To resolve this dilemma, modern information retrieval relies on a two-stage architecture. By combining a fast, high-recall bi-encoder with a slow, high-precision cross-encoder, these pipelines achieve both sub-second latency and fine-grained semantic accuracy.


The Core Dilemma: Representation vs. Interaction

To understand why a two-stage pipeline works, we must first look at how bi-encoders and cross-encoders process text differently.

Stage 1: Bi-Encoders and the Power of Pre-Computation

A bi-encoder (or dual-encoder) processes the query and the document completely independently. It maps the query into a single vector space, and the document into the same vector space. Because the document text does not depend on the query, we can pre-compute embeddings for all millions of documents in our database offline and store them in an index.

$$\text{Score}(q, d) = \cos(\mathbf{E}_q, \mathbf{E}_d)$$

When a query arrives, the system only needs to encode the query once, and then perform a rapid vector similarity search (such as a dot product or cosine similarity) against the pre-computed document vectors. As highlighted by Gregor Geigle et al. in Retrieve Fast, Rerank Smart: Cooperative and Joint Approaches for Improved Cross-Modal Retrieval, this independent mapping allows vector-space search engines to reduce search latencies from minutes to milliseconds.

However, this speed comes at a cost known as the representation bottleneck. Because the document vector must be generated without knowing what the query will be, it has to compress the entire meaning of the document into a single fixed-size vector. It cannot perform token-to-token alignment between the query and the text.

Stage 2: Cross-Encoders and Full Cross-Attention

A cross-encoder eliminates the representation bottleneck by feeding the query and the document into the Transformer network together, separated by a special token (e.g., [CLS] Query [SEP] Document).

This allows every single token in the query to attend to every single token in the document via self-attention layers. The model can weigh specific words in the document directly against specific words in the query, capturing highly complex dependencies, negation, and context.

The catch is computational complexity. Because the document representation changes dynamically based on the query, no pre-computation is possible. We must run the entire deep Transformer network at inference time for every single query-document pair. If our database has 10 million documents, running 10 million forward passes through a Transformer for one search query is completely intractable.


The Solution: Chaining the Two Paradigms

The two-stage pipeline resolves this by treating retrieval as a funnel. Instead of choosing between speed and accuracy, the architecture uses the bi-encoder to solve the scale problem, and the cross-encoder to solve the precision problem.

[ Massive Corpus: Millions of Documents ]
                   │
                   ▼
       ┌───────────────────────┐
       │   Stage 1: Retriever  │  <-- Fast Vector Search (O(log N))
       │     (Bi-Encoder)      │
       └───────────────────────┘
                   │
                   ▼  [ Shortlist: Top K Candidates (e.g., K=100) ]
       ┌───────────────────────┐
       │   Stage 2: Re-ranker  │  <-- Deep Cross-Attention (O(K))
       │    (Cross-Encoder)    │
       └───────────────────────┘
                   │
                   ▼
         [ Final Ranked Results ]
  1. The Retrieval Stage (High Recall): The bi-encoder acts as a coarse filter. It scans the entire corpus of millions of items and discards 99.9% of the irrelevant data, returning a small shortlist of candidates (typically the top 50 to 100 items).
  2. The Re-ranking Stage (High Precision): The cross-encoder takes only those few dozen candidates and performs deep, computationally expensive cross-attention to determine the exact final order.

As Yifu Qiu et al. demonstrate in DuReader-Retrieval: A Large-scale Chinese Benchmark for Passage Retrieval from Web Search Engine, this combination provides a highly effective balance, allowing systems to capture fine-grained semantic interactions without sacrificing the rapid initial retrieval required by real-world applications.


Why It Works: Intuition Across Domains

The robustness of this two-stage mechanism is proven by its ubiquity across vastly different NLP and multi-modal tasks. The same core intuition—using a fast vector filter followed by a deep contextual verifier—recurs across several specialized domains:

1. Disambiguation in Entity and Event Linking

In entity linking, a system must map a text mention (like "Boston") to the correct entry in a massive knowledge base like Wikipedia. A bi-encoder can easily find candidates based on surface similarity, but it struggles with deep contextual ambiguity.

To solve this, Tiantian Zhu et al. in Enhancing Entity Representations with Prompt Learning for Biomedical Entity Linking utilize a fast bi-encoder for initial candidate retrieval, followed by a computationally intensive prompt-based ranker to handle complex disambiguation. Similarly, Xiaodong Yu et al. apply this dual-stage framework to event grounding in Event Linking: Grounding Event Mentions to Wikipedia, using a cross-encoder to perform joint processing of mentions and candidate titles to capture deep semantic interactions.

2. Domain-Specific Search and RAG

In specialized search domains, such as legal drafting or medical records, missing a subtle linguistic marker can ruin the utility of the system.

3. Handling Noisy or Evolving Data

When data contains errors, edits, or misspellings, independent vector representations often break down because the semantic distance shifts.

In Noise-Robust De-Duplication at Scale, Emily Silcock et al. demonstrate that a multi-stage pipeline allows a fast bi-encoder to handle large-scale document filtering, while a cross-encoder handles the final verification needed to distinguish true duplicates from highly similar text containing OCR errors. This pattern is mirrored in misinformation tracking; Jabez Magomere et al. show in When Claims Evolve: Evaluating and Enhancing the Robustness of Embedding Models Against Misinformation Edits that combining a high-recall bi-encoder with a high-precision cross-encoder provides the deep interaction modeling required to match edited or rewritten social media claims to existing fact-checks.


Architectural Variations

While the traditional bi-encoder to cross-encoder pipeline is the standard, researchers have introduced notable variations to optimize specific bottlenecks:

  • Bi-Encoder Only Pipelines: To avoid the computational overhead of cross-encoders entirely during re-ranking, Yeon Seonwoo et al. introduce a two-step bi-encoder approach in Two-Step Question Retrieval for Open-Domain QA. By specializing a second bi-encoder exclusively on the reduced candidate set, they achieve accuracy gains while preserving the lightning-fast inference speed of vector-space models.
  • Alignment Before Fusion: In Alignment Precedes Fusion: Open-Vocabulary Named Entity Recognition as Context-Type Semantic Matching, Zhuoran Jin et al. explicitly separate the pipeline tasks into two distinct conceptual phases: using a dual-encoder for broad semantic alignment in a shared space, followed by a cross-encoder for fine-grained fusion to model complex dependencies.

Ultimately, whether applied to e-commerce query matching (Jiri Gesi et al.), database question answering (Stefano Campese et al.), or address matching in logistics (André V. Duarte et al.), the two-stage pipeline works because it respects the physical constraints of computation. It delegates the heavy lifting of broad search to fast vector geometry, saving the expensive luxury of deep attention for the candidates that matter most.

Key Foundational Work

DuReader-Retrieval: A Large-scale Chinese Benchmark for Passage Retrieval from Web Search Engine (2022) — contributed 1 concept elements to this direction.
Two-Step Question Retrieval for Open-Domain QA (2022) — contributed 1 concept elements to this direction.
Retrieve Fast, Rerank Smart: Cooperative and Joint Approaches for Improved Cross-Modal Retrieval (2021) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_bi_encoder_cross_encoder_pipeline_f820e760,
  title = {bi-encoder cross-encoder pipeline},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/bi-encoder-cross-encoder-pipeline/rdr_f820e760dd494fe597a1b3e5d8b17671},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_bi_encoder_cross_encoder_pipeline_f820e760_v1,
  title = {bi-encoder cross-encoder pipeline},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/bi-encoder-cross-encoder-pipeline/rdr_f820e760dd494fe597a1b3e5d8b17671/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake