Quantum-Inspired Pairwise Ranking Aggregation via Active LLM Feedback

Quan Mai, Susan Gauch, Gibson NkhataarXiv 2026 · arXiv

TL;DR

LLM-IRA is a "quantum-inspired" framework that uses Large Language Models (LLMs) to efficiently fuse multiple search engines. By asking the LLM to judge only the most controversial document pairs, it learns optimal fusion weights with 10x fewer calls than standard LLM rerankers, matching state-of-the-art performance at a fraction of the cost.

What Problem Does This Paper Address?

In modern Information Retrieval (IR) and Retrieval-Augmented Generation (RAG), we often use multiple "base rankers"—like traditional keyword search (BM25), dense vector embeddings, and sparse neural retrievers. No single ranker is perfect, so we use Ranking Aggregation (RA) to fuse their results.

The challenge lies in the trade-off between cost and quality:

  1. Unsupervised methods (like Reciprocal Rank Fusion) are ultra-fast and free but "dumb"—they can't adapt to specific queries.
  2. Supervised methods (Learning-to-Rank) require massive amounts of human-labeled training data.
  3. LLM Rerankers (like RankGPT) are highly accurate but prohibitively expensive and slow, often requiring hundreds of calls per query to sort a single list.

The authors seek a "middle way": a method that uses the reasoning power of LLMs to guide fusion weights without the massive inference bill.

Methods

The authors introduce LLM-IRA, extending a "quantum-inspired" framework originally designed for image retrieval.

The Quantum Analogy

In this framework, a document's relevance isn't a fixed value; it's a quantum state. A ranker's score is treated as the probability that this state will "collapse" into a "relevant" observation when measured. Using Gleason’s Theorem, the authors mathematically justify that the optimal way to fuse these observations is a simple linear combination of scores.

The Active Feedback Loop

Instead of asking an LLM to rank everything, LLM-IRA uses an interactive loop:

  1. Identify Disagreement: It looks at the top documents and finds pairs where the base rankers disagree most (e.g., BM25 loves Doc A, but the Dense retriever prefers Doc B).
  2. Active LLM Query: It asks the LLM a simple pairwise question: "Given this query, is A or B more relevant?"
  3. Weight Update: Based on the LLM's "measurement," it updates the weights ($w_m$) for each base ranker. If a ranker consistently agrees with the LLM, its weight increases.
  4. Pointwise Fusion: Once the weights are learned (usually after ~25-30 calls), the final ranking is generated using a lightning-fast linear sum.

Datasets

The method was evaluated on several standard IR benchmarks:

  • TREC Deep Learning (2019 & 2020): Large-scale web search tasks.
  • BEIR Benchmark: A diverse set of "out-of-domain" tasks including Covid (medical), SciFact (scientific), DBpedia (entity search), and NFCorpus (nutrition).
  • Base Rankers: BM25, nomic-embed-text-v1.5 (dense), and SPLADE (sparse).

Key Insights and Ideas

  • Disagreement is the Signal: You don't need to label every document. The most valuable information for a fusion model lies in the "contentious zone"—the documents where your existing tools can't agree.
  • Efficiency Plateau: One of the most striking findings is that the model reaches peak performance very quickly. As shown in the paper's profiling, it often beats the best unsupervised baselines within just 5 LLM calls per query.
  • Judge Agnosticism: The system is robust to the choice of LLM. Moving from a smaller 8B model (Llama-3.1) to a larger 14B model (Qwen-2.5) yielded nearly identical results, suggesting that "judge scale" isn't the bottleneck for this task.

Key Results

LLM-IRA consistently outperformed strong unsupervised baselines and matched the performance of expensive state-of-the-art rerankers.

Table 1. NDCG@10 comparison across TREC DL and BEIR benchmarks, using nomic-embed- text-v1.5 as the dense retriever and BM25+Dense+SPLADE (𝑀=3) as base rankers throughout. LLM-IRA (Llama-3.1-8B) values use the final adopted configuration: contentious-zone 𝑟=40, 𝐾=3 pairs/round, and confidence-threshold 𝜏=2.5 on the LLM logit gap. Best overall in bold; best unsupervised underlined (ties shown on all tied cells). ∗Statistically significant improvement over the best unsupervised aggregator in that column (Wilcoxon signed-rank test, paired per-query NDCG@10, 𝑝< 0.05); see §6 for the full per-column significance breakdown. ‡The accumulated- refit row uses weight-sharpness 𝛽=3 and the accumulated-refit update rule described in §4 (vs. 𝛽=1/momentum for the row above). Its ∗markers follow the same convention as the momentum row; see §6 for the full significance breakdown, including comparisons against the momentum row and LTR-Logistic.

As seen in the cost-effectiveness comparison:

  • Cost Savings: Compared to PRP-Graph (a popular pairwise LLM reranker), LLM-IRA achieved higher effectiveness while using only 5.6% of the LLM calls and 2.7% of the prompt tokens.
  • Performance: It trailed the state-of-the-art Setwise-HeapSort by a negligible 0.47 NDCG points but did so at roughly 1/5th the cost.
Table 2. Cost–effectiveness comparison on TREC DL 2019, at the MiniLM/Flan-T5-Large tier (matching the LLM reranker baselines’ judge). ∗LLM reranker numbers reproduced from Wang et al. [31] using Flan-T5-Large. The LLM-IRA row uses the accumulated-refit update rule (𝛽=3) with BM25+Dense (MiniLM) base rankers (𝑀=2), contentious-zone 𝑟=20, and the default confidence threshold (𝜏=2.0); this secondary tier is retained for a judge-matched cost comparison, distinct from the primary nomic/Llama-3.1-8B tier in Table 1. LTR-Logistic is omitted here: it uses the identical disagreement-based sampling and LLM-call budget as LLM-IRA (∼27 calls/query), so its cost profile is the same; see Table 1 for its NDCG@10 comparison.

Limitations and Future Work

  • LLM Noise: If the LLM judge provides incorrect feedback on a "contentious" pair, it can heavily penalize a superior base ranker. The authors use a confidence-gating threshold to mitigate this, but it isn't foolproof.
  • Binary Constraints: The current quantum model only considers "A is better than B" or vice versa. It doesn't yet have a mathematical way to handle cases where both documents are completely irrelevant.
  • Pointwise Reversion: To keep inference fast, the final step reverts to a 1D linear weighting. The authors suggest that future work could explore scalable ways to maintain a strictly pairwise logic through to the final output.

Summary updated

Abstract

Ranking aggregation from multiple retrievers is a powerful yet under-explored technique in modern retrieval and RAG systems. We introduce LLM-IRA, the first extension of the quantum-inspired interactive ranking aggregation framework to document reranking. By leveraging large language models (LLMs) solely for pairwise judgments on high-disagreement document pairs, our method learns fusion weights in a principled, interactive way. The final ranking is a simple, ultra-fast pointwise linear fusion of scores, no full pairwise matrix or expensive listwise prompting is required. Experiments on the BEIR benchmark and TREC Deep Learning tracks with an open-weight instruction-tuned judge (Llama-3.1-8B-Instruct) show that LLM-IRA matches or numerically exceeds strong static fusion baselines on 5 of 6 evaluated benchmarks. We report two weight-update variants: momentum blending and a full accumulated-history refit, which together reach statistical significance (Wilcoxon signed-rank, p<0.05) over the best unsupervised aggregator on 4 of 6 benchmarks (DL19 under momentum; DL20, DBpedia, and NFCorpus under accumulated refit, which is the stronger of the two configurations overall). On the benchmarks we profiled (DL19, DL20), it already clears the best unsupervised baseline within as few as 5 LLM calls per query, well before exhausting its typical ~25--30 call budget. LLM-IRA also achieves cost-effectiveness comparable to state-of-the-art zero-shot LLM rerankers at an order of magnitude fewer LLM calls, and requires no training data or model fine-tuning.
BibTeX
@misc{mai2026quantum,
  author = {Quan Mai and Susan Gauch and Gibson Nkhata},
  title = {{Quantum-Inspired Pairwise Ranking Aggregation via Active LLM Feedback}},
  year = {2026},
  eprint = {2610.38683},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.38683},
}
Figure