Active Learning of Approximate LLM Ranking for Energy-Efficient Retrieval of Long Queries

Ohad Porat, Elad Yom-TovarXiv 2026 · arXiv

TL;DR

The LLM Active Approximation Model (LAAM) is a hybrid retrieval framework that uses active learning to train a lightweight "surrogate" model to mimic expensive Large Language Model (LLM) ranking behavior. For complex, verbose queries, LAAM achieves roughly 87% of the accuracy of full LLM reranking while consuming only 3.3% of the energy, offering a tunable "dial" between computational cost and retrieval quality.

What Problem Does This Paper Address?

Modern Information Retrieval (IR) faces a stark tension between efficiency and semantic depth. Traditional lexical retrievers like BM25 are incredibly fast and energy-efficient, but they struggle with "verbose" queries—long, context-rich descriptions of an information need—because they rely on simple keyword matching.

Large Language Models (LLMs) solve this by capturing nuanced semantic meaning, but they are computationally and energetically ruinous. Reranking hundreds of documents per query with an LLM can consume orders of magnitude more energy than a standard search. The authors address this "Green IR" challenge by asking: Can we get the semantic benefits of an LLM without paying the full energy tax?

Methods

The authors propose LAAM, a framework that treats the LLM as an expensive "teacher" and a lightweight model (Gradient Boosting) as a "student" or surrogate.

The process, illustrated in Figure 1, works as follows:

  1. Initial Retrieval: A standard lexical retriever (Lucene/BM25) generates a candidate set.
  2. Feature Extraction: For each query-document pair, the system extracts a 770-dimensional feature vector combining semantic signals (sentence-transformer embeddings) and lexical signals (TF-IDF similarity and term overlap).
  3. Active Learning Loop: Instead of labeling every document with the LLM, LAAM uses an active learning strategy. It selects a small budget of documents ($K$) to be scored by the LLM.
  4. Greedy Selection: The system prioritizes documents where the current surrogate model's prediction most strongly disagrees with the initial lexical score. The rationale is to target documents where the "semantic gap" is largest—where keyword matching is likely failing.
  5. Incremental Refinement: The surrogate model is updated after each LLM evaluation, progressively aligning itself with the LLM's judgment.
Figure 1: Architecture of the LAAM-based reranking retrieval pipeline.

Datasets

The framework was evaluated using the TREC Robust 2004 dataset, specifically topics 301–350. The authors tested three levels of query verbosity provided by the dataset:

  • Title: Short, keyword-based queries.
  • Description: Medium-length queries clarifying intent.
  • Narrative: Long, detailed queries defining specific relevance criteria.

DeepSeek v3 was utilized as the "gold standard" LLM ranker for the experiments.

Key Insights and Ideas

  • Energy as a First-Class Metric: The paper explicitly measures energy consumption in Joules (or kJ/MJ) alongside traditional accuracy metrics like P@10 and MAP. This framing treats computational budget as a controllable resource rather than a fixed cost.
  • The Power of Disagreement: The "Greedy" selection policy (picking documents with maximal disagreement between the surrogate and the lexical prior) significantly outperformed random sampling. This suggests that the most valuable information for the model lies in the documents that the lexical retriever "misunderstands."
  • Diminishing Returns of LLM Calls: The energy-accuracy curve is non-linear. Most retrieval gains are captured within the first few dozen LLM evaluations. Doubling the LLM budget ($K=50$ to $K=100$) doubles the energy cost but yields only modest accuracy improvements.
  • Verbosity Sensitivity: The hybrid approach is most effective for long, "narrative" queries. For short "title" queries, BM25 is often "good enough," and the overhead of semantic reranking provides less marginal benefit.

Key Results

For verbose Narrative queries:

  • Performance: LAAM ($K=100$) achieved a P@10 of 0.459, compared to 0.526 for full DeepSeek reranking and 0.243 for the BM25 baseline.
  • Efficiency: LAAM reached ~87% of the LLM's accuracy while using only 3.3% of the energy (430 kJ vs. 12.9 MJ per query).
  • Statistical Significance: LAAM was found to be statistically indistinguishable from exhaustive LLM reranking in terms of Precision at 1 and 10, though the full LLM still held a significant lead in Mean Average Precision (MAP).

Limitations and Future Work

  • Benchmark Leakage: The authors acknowledge that TREC Robust 2004 is a public dataset. There is a risk that DeepSeek's high performance stems partly from having seen these documents/judgments during training.
  • Energy Estimation: While LAAM's energy was measured at the hardware level, DeepSeek's energy was estimated via API statistics and infrastructure models, which may not be perfectly precise.
  • Query Independence: Currently, LAAM is trained from scratch for every new query. The authors suggest that transfer learning—using knowledge from previous queries to initialize the surrogate—could further reduce the number of LLM calls needed.
  • RAG Integration: A promising future direction is applying this architecture to Retrieval-Augmented Generation (RAG) systems to prune documents before they reach the generation stage, saving both cost and context window space.

Summary updated

Abstract

Search engines and Large Language Models (LLMs) offer complementary strengths for document retrieval: lexical retrievers which rely on keyword matching are fast and energy-efficient, whereas LLMs capture semantic meaning and excel at interpreting long, context-rich queries. However, this comes at a cost that is computationally and energetically prohibitive. We propose the LLM Active Approximation Model (LAAM), a hybrid retrieval framework that introduces a lightweight learned predictor which serves as an intermediary between semantic retrieval and LLM retrieval. LAAM is refined through an active learning loop where an LLM labels a small number of documents. By concentrating LLM evaluations on the most informative documents, the framework substantially reduces inference cost while preserving retrieval effectiveness. We evaluate the proposed framework on TREC Robust 2004 using Lucene with BM25 retrieval and DeepSeek v3 as the LLM, under three levels of query verbosity. For verbose queries, LAAM achieves 87% of average P@10 achieved by DeepSeek's reranking (0.459 vs. 0.526), while requiring only 3.3% of the per-query energy consumption. LAAM is statistically significantly better than BM25 and is indistinguishable from DeepSeek on average precision, though not on MAP (Friedman test, P<0.05 with post-hoc testing). Overall, our results demonstrate that selectively approximating LLM relevance judgments through active learning enables retrieval accuracy to be controlled directly through the evaluation (and consequently energy) budget. Rather than committing to either inexpensive (and less accurate) lexical retrieval or exhaustive (and expensive) LLM reranking, our framework exposes a continuous spectrum of operating points along the energy--accuracy curve, allowing practitioners to select how to trade retrieval quality with computational budget.
BibTeX
@misc{porat2026active,
  author = {Ohad Porat and Elad Yom-Tov},
  title = {{Active Learning of Approximate LLM Ranking for Energy-Efficient Retrieval of Long Queries}},
  year = {2026},
  eprint = {2611.16488},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.16488},
}
Figure