Can LLM Rerankers Predict Their Own Ranking Performance?

Shiyu Ni, Keping Bi, Jiafeng Guo, Jingtong Wu, Zengxin Han, Xueqi ChengarXiv 2026 · arXiv

TL;DR

Can an LLM search reranker accurately predict when its own search results are likely to be incorrect? This paper explores reranker-internal Query Performance Prediction (QPP), demonstrating that while standard LLMs are wildly overconfident, we can extract highly accurate, well-calibrated performance predictions using metric-specific self-consistency or lightweight supervised fine-tuning.

What Problem Does This Paper Address?

In information retrieval, search effectiveness varies dramatically from query to query. Some queries are clear and well-covered by a document collection, while others are ambiguous or suffer from vocabulary mismatch. When a search system returns poor results, it can mislead users or degrade the performance of downstream applications like Retrieval-Augmented Generation (RAG).

Query Performance Prediction (QPP) aims to estimate the quality of a search result list without needing human ground-truth labels. However, traditional QPP methods rely on external, separate modules applied after the ranking process. The authors of this paper ask a compelling question: Can a sequence-to-sequence LLM reranker evaluate the quality of the ranking it has just produced? If a model understands relevance well enough to rank documents, it should inherently possess the signals needed to judge its own reliability.

Figure 1: Overview of reranker-internal QPP. In this example, the model incorrectly ranks the less relevant Passage 2 ahead of Passage 1. We expect the reranker not only generates the ranked list but also expresses its confidence, helping prevent users from being misled by an erroneous ranking result.

Methods

The authors investigate both training-free and training-based approaches to enable internal QPP:

1. Training-Free Approaches

  • Direct Verbalized Confidence: The LLM is simply prompted to output a numerical confidence score (e.g., "Confidence: 0.85") alongside its ranked list.
  • Metric-Specific Self-Consistency: The model generates multiple ranking lists by sampling at a higher temperature ($T=1$). The authors then measure how consistent these sampled lists are against a stable, greedily decoded ranking ($T=0$). Crucially, they tailor the consistency checks to specific evaluation metrics:
  • Precision@k: Measured by the simple overlap ratio of the top-$k$ items.
  • NDCG@k: Measured using Rank Biased Overlap (RBO) to account for position-based discounting.
  • MRR@k: Evaluated based on the reciprocal rank of the first list's top item within the second list.

2. Training-Based (Supervised) Approaches

To fix the calibration errors of raw LLMs, the authors propose two fine-tuning methods using a multi-task loss function that balances ranking capability with confidence accuracy:

  • Verb-Num: The LLM is trained to output a single scalar value representing a target metric (like normalized DCG@10).
  • Verb-List: The LLM outputs an ordered list of binary indicators (e.g., [1, 0, 1, 0...]) representing the predicted relevance of each top-ranked document. This approach is metric-agnostic and lets the model focus on simple relevance judgments rather than complex metric math.
Figure 2: Overview of ranking lists generation, QPP-Gen, metric-specific consistency checking, and verbalized ranking-quality estimates calibration. QPP-Gen uses an LLM for pointwise relevance judgments. Self-consistency- based confidence measures agreement between multiple ranking lists under different IR metrics. Verbalized train- ing labels are obtained by assessing rankings with both human annotation and LLM annotation.

Datasets

  • Training: The MS MARCO passage ranking dataset (20k balanced queries). Because human annotations in MS MARCO are sparse, the authors used a powerful reasoning model (Qwen3-32B) to augment the dataset with denser relevance labels.
  • Evaluation: Standard Information Retrieval benchmarks from the TREC Deep Learning (TREC-DL) tracks spanning 2019 to 2022.

Key Insights and Ideas

  • LLMs are Inherent Braggarts: Without training, asking an LLM to state its confidence directly results in severe overconfidence. Across multiple model sizes, raw verbalized confidence scores routinely exceeded 0.8, completely diverging from actual ranking performance.
  • Consistency is a Great Proxy for Quality: Sampling multiple ranking paths (self-consistency) provides a remarkably strong, training-free signal. If the LLM repeatedly ranks the same documents highly across different runs, it is highly likely to be correct.
  • The "Discriminability vs. Calibration" Trade-off: When fine-tuning, Verb-Num achieves better discriminability (higher Spearman correlation), meaning it is excellent at sorting which queries were hard versus easy. However, Verb-List achieves much better absolute calibration (lower Expected Calibration Error) because it breaks the problem down into token-by-token relevance judgments rather than forcing the model to calculate an abstract global score.

Key Results

  • Self-Consistency Beats the SOTA Unsupervised Baseline: In training-free settings, the proposed metric-specific self-consistency method achieved competitive Spearman correlations and significantly lower Expected Calibration Error (ECE) compared to QPP-Gen (the previous state-of-the-art unsupervised method). For instance, on Qwen2.5-14B, self-consistency dropped the NDCG-I@10 ECE from 0.208 to 0.158.
  • Supervised Methods Achieve SOTA: Both Verb-Num and Verb-List outperformed all existing supervised baselines (like BERTQPP and qppBERT-PL). On the TREC DL tracks, Verb-Num achieved an average Spearman correlation of up to 0.596, while Verb-List achieved a remarkably low average ECE of 0.179, all while adding negligible token generation overhead.

Limitations and Future Work

  • Binary Relevance Simplification: The study binarized relevance labels (relevant vs. irrelevant). Real-world search often relies on multi-level relevance (e.g., perfectly relevant, partially relevant), which remains unexplored here.
  • Annotation Bias: The supervised models suffered from slight overconfidence inherited from the data augmentation step, as the teacher model (Qwen3-32B) occasionally misclassified irrelevant documents as relevant.
  • Efficiency vs. Performance: While self-consistency is training-free, it requires generating 20 sampled ranking lists, making it computationally expensive.

Future work will focus on improving unsupervised internal QPP efficiency and exploring whether long-horizon reasoning (like chain-of-thought reflection) can help LLM rerankers better understand their own knowledge boundaries.

Summary updated

Abstract

Retrieval effectiveness varies substantially across queries, making it important to estimate ranking quality before relevance judgments are available. Query performance prediction (QPP) addresses this need, but most existing methods rely on external predictors after retrieval or reranking. In this paper, we study \textit{reranker-internal QPP}: can an LLM reranker estimate the quality of the ranking it has just produced? We investigate both training-free and training-based approaches. For training-free estimation, we examine metric-specific self-consistency across sampled rankings and verbalized confidence produced directly by the reranker. Experiments on TREC Deep Learning 2019--2022 with four LLMs show that self-consistency is competitive with the state-of-the-art (SOTA) approach and better calibrated in almost all settings, while direct verbalized confidence is severely overconfident. To improve verbalized confidence, we propose two supervised methods, Verb-Num and Verb-List, which enable LLM rerankers to produce calibrated ranking-quality estimates with only a few additional output tokens.
BibTeX
@misc{ni2026can,
  author = {Shiyu Ni and Keping Bi and Jiafeng Guo and Jingtong Wu and Zengxin Han and Xueqi Cheng},
  title = {{Can LLM Rerankers Predict Their Own Ranking Performance?}},
  year = {2026},
  eprint = {2606.03535},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2606.03535},
}

Loading related content...

Figure