Abstract
In this work, we present a systematic and comprehensive empirical evaluation of state-of-the-art reranking methods, encompassing large language model (LLM)-based, lightweight contextual, and zero-shot approaches, with respect to their performance in information retrieval tasks. We evaluate in total 22 methods, including 40 variants across several established benchmarks, including TREC DL19, DL20, and BEIR, as well as a novel dataset called FutureQueryEval, which is designed to test queries unseen by pretrained models. Our primary goal is to determine whether a performance disparity exists between LLM-based rerankers and their lightweight counterparts, particularly on novel queries, and to elucidate the underlying causes of any observed differences. Our findings indicate that while LLM-based rerankers demonstrate superior performance on familiar queries, their generalization ability to novel queries varies, with lightweight models offering comparable efficiency. We further identify that the novelty of queries significantly impacts reranking effectiveness, highlighting limitations in existing approaches.
Introduction
Modern Information Retrieval (IR) systems typically employ a two-stage pipeline: a rapid initial retriever (e.g., BM25) identifies a candidate set of documents, which a more sophisticated "reranker" then re-orders for optimal relevance. Recently, Large Language Models (LLMs) have emerged as the state-of-the-art for this second stage. However, the reliance on standard benchmarks like TREC or BEIR raises concerns regarding data contamination, as these public datasets were likely included in the training corpora of models like GPT-4. This study investigates whether LLMs possess true ranking capabilities or are merely recalling seen data, while also assessing the efficiency trade-offs against lightweight models.
Methodology
The study categorizes reranking into three distinct algorithmic paradigms based on how they process query-document relationships:
- Pointwise: Each query-document pair is scored independently (e.g., MonoT5). This approach is highly efficient ($O(n)$) but lacks context regarding the relative relevance of other documents.
- Pairwise: The model compares pairs of documents to determine which is more relevant (e.g., PRP). While more precise, it can be computationally expensive, reaching $O(n^2)$ complexity.
- Listwise: The model processes the query and the entire document list simultaneously (e.g., RankGPT). This allows for a holistic view of the candidate set but is constrained by the model's context window.
To ensure a fair comparison, the authors evaluated 22 methods (40 variants) using a unified framework called Rankify.
Experimental Setup
The evaluation utilized a comprehensive suite of benchmarks:
- Standard Benchmarks: TREC DL19, DL20, and eight subsets of the BEIR benchmark (e.g., SciFact, Covid).
- Open-Domain QA: Natural Questions (NQ) and WebQuestions.
- FutureQueryEval: A novel benchmark consisting of 148 queries and ~2,800 documents collected after April 2025. This dataset serves as a "contamination-free" zone to test the true generalization of models trained on data with earlier cutoffs.
Key Insights and Results
The Generalization Gap
A significant "temporal generalization gap" was observed across all model categories. Performance dropped by 5–15% on the FutureQueryEval dataset compared to standard benchmarks. This suggests that current LLM success in reranking is partially attributed to familiarity with existing benchmark data.
Robustness and Efficiency
Listwise models, such as Zephyr-7B, demonstrated the highest robustness to novelty, with the smallest performance degradation (~8%). Modeling inter-document relationships appears to provide a buffer when document content is unfamiliar. However, this robustness comes at a high computational cost; some listwise LLM approaches required over 50 minutes to process a single query set on an A100 GPU.
On standard benchmarks, RankGPT (GPT-4) and InRanker-3B were the top performers. For production environments, MonoT5-3B and Twolar-xl were identified as "sweet spots," providing strong generalization and performance without the extreme latency of GPT-4-based listwise methods.
Conclusion
The empirical evidence suggests that while LLMs are powerful rerankers, their performance is sensitive to query novelty and domain (struggling particularly with argumentative and informal text). The future of scalable, effective reranking likely involves hybrid strategies: utilizing efficient pointwise models for initial broad reranking and employing listwise LLM reasoning for the most complex or novel queries.
Summary updated