Introduction
In modern information retrieval, balancing efficiency and effectiveness remains a primary challenge. While simple models like BM25 can search billions of documents in milliseconds, they lack the semantic depth of Large Language Models (LLMs). Conversely, LLMs are highly effective but computationally prohibitive for large-scale initial retrieval. The "Naverloo" team—a collaboration between Naver Labs Europe and the University of Waterloo—addressed this by implementing a four-stage cascading architecture for the TREC 2023 Deep Learning and NeuCLIR tracks.
The Cascading Pipeline: "Zero, One, Two, Three"
The core of the Naverloo approach is a pipeline that progressively narrows the candidate pool using models of increasing complexity, playfully dubbed "Zero, One, Two, Three":
- Stage 0: First Stage Retrieval (Dual Encoders): A "wide net" approach using a hybrid of traditional sparse retrieval (BM25), learned sparse models (SPLADE), and dense retrievers (AggRetriever) to identify the top 1,000 candidates.
- Stage 1: Second Stage Ranker (mono): Pointwise reranking using cross-encoders like DeBERTa and RankT5 to score individual query-document pairs.
- Stage 2: Third Stage Ranker (duo): Pairwise reranking using DuoT5. Due to the quadratic complexity of comparing document pairs, this stage is restricted to the top 50 documents.
- Stage 3: Fourth Stage Ranker (listo): Listwise reranking using RankGPT (GPT-4). This stage processes the top 30 results simultaneously to produce the final ordering.
The team utilized the ranx framework for ensembling scores at each stage, enhancing the robustness of the final rankings.
Experimental Results
The system was evaluated on the TREC Deep Learning (DL) track for English passage and document retrieval, and the NeuCLIR track for cross-language retrieval in Persian, Russian, and Chinese.
TREC Deep Learning Track
In the passage ranking task, the cascading architecture demonstrated clear incremental gains at each stage. The final listo stage achieved an nDCG@10 of 0.6994, significantly outperforming the track median of 0.5329.
NeuCLIR Track
The "0-1-2-3" strategy proved equally effective for multilingual retrieval. In the Persian task, the pipeline reached an nDCG of 0.7020, demonstrating that the cascading hierarchy remains robust across translation layers and diverse languages.
Key Insights and Discussion
The Naverloo team identified several critical trends in modern retrieval:
- The Widening Performance Gap: There is a significant performance gap between initial dual-encoder retrieval and final LLM-based reranking. In 2023, first-stage retrievers are no longer sufficient on their own to reach top-tier performance.
- Reproducibility Concerns: While GPT-4 provides state-of-the-art results, its proprietary and non-deterministic nature poses a challenge to scientific reproducibility, making it difficult to get the exact same ranking twice.
- Ensembling vs. Strong Baselines: While ensembling multiple retrievers provides marginal gains, a single well-trained sparse model like SPLADE remains a highly competitive and more efficient alternative.
Conclusion
The Naverloo team's success at TREC 2023 highlights the power of cascading architectures. By combining the speed of dual encoders with the sophisticated reasoning of LLMs, they achieved state-of-the-art results across English and multilingual benchmarks. Future work will focus on exploring open-source alternatives to GPT-4 and investigating more efficient single-stage models that can match the performance of complex pipelines.
Summary updated