Modern information retrieval systems are almost always built as a hierarchy. Because evaluating a complex neural model over millions of documents is computationally impossible in real-time, systems use a "cascade" or "multi-stage" architecture: a fast, coarse-grained retriever (like BM25 or a bi-encoder) narrows the field to a few hundred candidates, which a slow, high-capacity ranker (like a Cross-Encoder or an LLM) then re-orders.
While this approach is efficient, it introduces a fundamental structural flaw. As noted by Andrew Yates and Samarth Mehrotra in MPII at TREC CAsT 2019: Incoporating Query Context into a BERT Re-ranker, the final output quality of these systems is strictly capped by the recall of the very first stage. If the initial heuristic fails to surface a relevant document, the sophisticated downstream model never sees it, and the system has no way to recover that lost information.
The Hard Ceiling of Initial Recall
In a two-stage system, the second stage can only improve precision—it can move the "right" answer to the top of the list, but it cannot find an answer that isn't in the list. This creates what researchers call a recall bottleneck.
Jiafeng Guo and colleagues argue in Beyond Precision: A Study on Recall of Initial Retrieval with Neural Representations that while the field has historically obsessed over re-ranking precision, the real performance ceiling is determined by the first-pass retriever. This bottleneck is particularly acute when there is a "vocabulary mismatch"—where the user's query uses different words than the relevant document. Traditional keyword-based retrievers like BM25 often fail here, permanently excluding relevant results before a neural model like BERT even gets to look at them.
This dependency is a recurring theme across diverse domains:
- Conversational Search: Nikos Voskarides and Andreas Panteli found that if the first stage cannot resolve conversational context (like "it" or "that"), the entire pipeline fails because the correct passages are never retrieved (ILPS at TREC 2019 Conversational Assistant Track).
- Multimodal Retrieval: In tasks like Composed Image Retrieval, Weixuan Sun and Damien Teney demonstrate that even the most powerful reasoning models are useless if the target image is pruned during the initial coarse filtering (Candidate Set Re-ranking for Composed Image Retrieval with Dual Multi-modal Encoder).
- Robustness: Shengyao Zhuang and Guido Zuccon showed that query typos cause a "cascading failure" where the first stage misses the document, leaving the re-ranker with nothing relevant to process (Dealing with Typos for BERT-based Passage Retrieval and Ranking).
Moving Neural Intelligence Upstream
To break this bottleneck, research has shifted toward making the first stage "smarter" without sacrificing its speed. One dominant strategy is Dense Retrieval, where documents are encoded into vectors offline. Jingtao Zhan and colleagues demonstrated with RepBERT: Contextualized Text Embeddings for First-Stage Retrieval that using BERT-based embeddings at the very first stage can capture semantic matches that keywords miss, significantly raising the recall ceiling.
Another approach involves Mutual Learning, where the retriever and the ranker are trained together. Chongyang Tao and Jiazhan Feng proposed a framework where a sophisticated Cross-encoder "teaches" a fast Bi-encoder retriever which documents are actually relevant, ensuring the first stage is optimized specifically to serve the second stage (Building an Efficient and Effective Retrieval-based Dialogue System via Mutual Learning).
Approximating the Gold Standard
The "Gold Standard" for ranking is the Cross-Encoder, which looks at the query and document simultaneously. However, Cross-Encoders are too slow to be used as retrievers. To bypass the two-stage bottleneck entirely, researchers like Nishant Yadav, Nicholas Monath, and Manzil Zaheer have developed methods like ADACUR and ANNCUR.
These methods use matrix factorization (specifically CUR decomposition) to approximate Cross-Encoder scores efficiently. By iteratively "zooming in" on relevant items, they allow a system to behave like a high-precision ranker while maintaining the speed of a retriever, effectively dissolving the boundary between the two stages and eliminating the traditional recall bottleneck (Efficient k-NN Search with Cross-Encoders using Adaptive Multi-Round CUR Decomposition).
Ultimately, the trend in the field is a move away from "filtering then thinking" toward "thinking while searching." Whether through neural first-stage indexes or adaptive matrix approximations, the goal is to ensure that the "gatekeeper" of the retrieval pipeline is just as capable as the models that follow it.