TL;DR
Can an LLM search reranker accurately predict when its own search results are likely to be incorrect? This paper explores reranker-internal Query Performance Prediction (QPP), demonstrating that while standard LLMs are wildly overconfident, we can extract highly accurate, well-calibrated performance predictions using metric-specific self-consistency or lightweight supervised fine-tuning.
What Problem Does This Paper Address?
In information retrieval, search effectiveness varies dramatically from query to query. Some queries are clear and well-covered by a document collection, while others are ambiguous or suffer from vocabulary mismatch. When a search system returns poor results, it can mislead users or degrade the performance of downstream applications like Retrieval-Augmented Generation (RAG).
Query Performance Prediction (QPP) aims to estimate the quality of a search result list without needing human ground-truth labels. However, traditional QPP methods rely on external, separate modules applied after the ranking process. The authors of this paper ask a compelling question: Can a sequence-to-sequence LLM reranker evaluate the quality of the ranking it has just produced? If a model understands relevance well enough to rank documents, it should inherently possess the signals needed to judge its own reliability.
Methods
The authors investigate both training-free and training-based approaches to enable internal QPP:
1. Training-Free Approaches
- Direct Verbalized Confidence: The LLM is simply prompted to output a numerical confidence score (e.g., "Confidence: 0.85") alongside its ranked list.
- Metric-Specific Self-Consistency: The model generates multiple ranking lists by sampling at a higher temperature ($T=1$). The authors then measure how consistent these sampled lists are against a stable, greedily decoded ranking ($T=0$). Crucially, they tailor the consistency checks to specific evaluation metrics:
- Precision@k: Measured by the simple overlap ratio of the top-$k$ items.
- NDCG@k: Measured using Rank Biased Overlap (RBO) to account for position-based discounting.
- MRR@k: Evaluated based on the reciprocal rank of the first list's top item within the second list.
2. Training-Based (Supervised) Approaches
To fix the calibration errors of raw LLMs, the authors propose two fine-tuning methods using a multi-task loss function that balances ranking capability with confidence accuracy:
- Verb-Num: The LLM is trained to output a single scalar value representing a target metric (like normalized DCG@10).
- Verb-List: The LLM outputs an ordered list of binary indicators (e.g.,
[1, 0, 1, 0...]) representing the predicted relevance of each top-ranked document. This approach is metric-agnostic and lets the model focus on simple relevance judgments rather than complex metric math.
Datasets
- Training: The MS MARCO passage ranking dataset (20k balanced queries). Because human annotations in MS MARCO are sparse, the authors used a powerful reasoning model (Qwen3-32B) to augment the dataset with denser relevance labels.
- Evaluation: Standard Information Retrieval benchmarks from the TREC Deep Learning (TREC-DL) tracks spanning 2019 to 2022.
Key Insights and Ideas
- LLMs are Inherent Braggarts: Without training, asking an LLM to state its confidence directly results in severe overconfidence. Across multiple model sizes, raw verbalized confidence scores routinely exceeded 0.8, completely diverging from actual ranking performance.
- Consistency is a Great Proxy for Quality: Sampling multiple ranking paths (self-consistency) provides a remarkably strong, training-free signal. If the LLM repeatedly ranks the same documents highly across different runs, it is highly likely to be correct.
- The "Discriminability vs. Calibration" Trade-off: When fine-tuning, Verb-Num achieves better discriminability (higher Spearman correlation), meaning it is excellent at sorting which queries were hard versus easy. However, Verb-List achieves much better absolute calibration (lower Expected Calibration Error) because it breaks the problem down into token-by-token relevance judgments rather than forcing the model to calculate an abstract global score.
Key Results
- Self-Consistency Beats the SOTA Unsupervised Baseline: In training-free settings, the proposed metric-specific self-consistency method achieved competitive Spearman correlations and significantly lower Expected Calibration Error (ECE) compared to QPP-Gen (the previous state-of-the-art unsupervised method). For instance, on Qwen2.5-14B, self-consistency dropped the NDCG-I@10 ECE from 0.208 to 0.158.
- Supervised Methods Achieve SOTA: Both Verb-Num and Verb-List outperformed all existing supervised baselines (like BERTQPP and qppBERT-PL). On the TREC DL tracks, Verb-Num achieved an average Spearman correlation of up to 0.596, while Verb-List achieved a remarkably low average ECE of 0.179, all while adding negligible token generation overhead.
Limitations and Future Work
- Binary Relevance Simplification: The study binarized relevance labels (relevant vs. irrelevant). Real-world search often relies on multi-level relevance (e.g., perfectly relevant, partially relevant), which remains unexplored here.
- Annotation Bias: The supervised models suffered from slight overconfidence inherited from the data augmentation step, as the teacher model (Qwen3-32B) occasionally misclassified irrelevant documents as relevant.
- Efficiency vs. Performance: While self-consistency is training-free, it requires generating 20 sampled ranking lists, making it computationally expensive.
Future work will focus on improving unsupervised internal QPP efficiency and exploring whether long-horizon reasoning (like chain-of-thought reflection) can help LLM rerankers better understand their own knowledge boundaries.
Summary updated