The bedrock of information retrieval (IR) evaluation is the Cranfield paradigm: a fixed collection of documents, a set of search queries, and a comprehensive set of relevance judgments mapping queries to documents. Historically, creating these benchmarks required human experts to manually read and label thousands of query-document pairs.
As web corpora have scaled to billions of documents and user needs have grown increasingly dynamic, this manual approach has hit a hard scalability bottleneck. Exhaustive human annotation across massive datasets is logistically impossible, as noted by David Wadden in the context of open-domain verification frameworks like SciFact-Open: Towards open-domain scientific claim verification (emnlp 2022). When human labeling is constrained to a small subset of documents (pooling), it creates "holes" in the dataset. If a new, high-performing retrieval system finds a perfectly valid document that was not part of the original manual evaluation pool, standard metrics penalize the system as if it retrieved garbage.
To solve this crisis, researchers have developed methods to scale relevance assessments without relying on continuous, expensive human intervention. These approaches operate on a shared intuition: we can bypass the manual annotation bottleneck by substituting explicit expert judgments with implicit behavioral signals, statistical consensus, or automated semantic models.
Extracting Relevance from User Behavior
The most abundant, zero-cost alternative to manual labeling is the trail of data users leave behind while searching. Every click, scroll, and mouse movement contains a hidden signal about document relevance.
However, user feedback is notoriously noisy. A user might click a document simply because it is the first result on the page (position bias) or because it has an enticing title, only to realize the content is irrelevant. To turn this messy data into reliable evaluation metrics, researchers model the relationship between user actions and true relevance.
[User Query] ──> [Search Results] ──> [Position Bias / Presentation]
│
▼
[Observed Click]
│
(De-biasing Model)
│
▼
[Inferred Relevance]
In The Value of User Feedback (ECIR 2011), Thorsten Joachims argues that search engines can treat the interaction loop between users and systems as a continuous stream of weak supervision. By analyzing large-scale search logs, systems can optimize ranking functions online without needing static, offline human judgments, a paradigm expanded upon by Katja Hofmann in Search engines that learn online (sigir 2011).
To make these behavioral signals mathematically rigorous, click models isolate the probability of a user examining a document from the probability of them clicking it. As shown by Ben Carterette and Rosie Jones in Evaluating Search Engines by Modeling the Relationship Between Relevance and Clicks (neurips 2007), when we aggregate millions of these interactions, the random noise of individual user behavior cancels out, leaving a highly accurate reflection of system quality.
Going beyond simple clicks, the authors of A Mouse-Trajectory Based Model for Predicting Query-URL Relevance (aaai 2012) incorporate fine-grained mouse movements—such as hesitation or reading patterns—to infer multi-level relevance labels even for rare queries that suffer from data sparsity.
System Consensus and Statistical Sampling
If we lack user logs and cannot afford human judges, how do we evaluate a set of retrieval systems? A powerful intuition is to let the systems evaluate themselves through consensus.
Imagine submitting a query to ten different, independently designed search engines. If eight of those systems return Document A in their top three results, but only one returns Document B, it is highly probable that Document A is more relevant than Document B.
System 1: [Doc A, Doc C, Doc D]
System 2: [Doc B, Doc A, Doc E] ───> Consensus: Doc A appears most frequently
System 3: [Doc F, Doc A, Doc C] at high ranks.
Tetsuya Sakai validates this approach in Ranking Retrieval Systems without Relevance Assessments: Revisited (EVIA@NTCIR 2010), demonstrating that a simple majority vote among competing retrieval systems can reliably forecast final system rankings. Similarly, the authors of Using Clustering to Improve Retrieval Evaluation without Relevance Judgments (coling 2010) refine this concept by clustering top-ranked search results and selecting representative documents, preventing a single dominant system architecture from biasing the consensus.
When some human annotation budget is available, the goal shifts to maximizing its efficiency through intelligent sampling. Instead of evaluating every query exhaustively, frameworks can treat evaluation as an optimization problem. In An uncertainty-aware query selection model for evaluation of IR systems (sigir 2012), the authors introduce an iterative algorithm that selects a minimal, high-uncertainty subset of queries for human review, ensuring that the resulting system ranking remains statistically stable at a fraction of the cost.
Large Language Models as Silver-Standard Judges
The emergence of Large Language Models (LLMs) has introduced a third paradigm: replacing human judges with automated semantic evaluators. If an LLM can understand the nuances of natural language, it can read a query and a document and output a relevance score.
The transition to model-based evaluation drastically lowers financial and temporal barriers, as highlighted by Jheng-Hong Yang and Jimmy Lin in Toward Automatic Relevance Judgment using Vision-Language Models for Image-Text Retrieval Evaluation (sigir 2024). In large-scale benchmarks like the TREC Deep Learning Track, the sheer volume of relevant documents in massive corpora now exceeds human capacity, making automated prompting approaches essential for preventing score saturation, as detailed by Nick Craswell and Bhaskar Mitra in Overview of the TREC 2023 Deep Learning Track (TREC 2023).
Why do LLM judges work so well for evaluation even when their individual labels do not perfectly match human choices? The key lies in the distinction between absolute label accuracy and relative system ranking consistency.
Query-Doc Pair Human Label LLM Label System A (Retrieves 1 & 2)
Pair 1 Perfect Highly Human Score: High
Pair 2 Partial Relevant LLM Score: High
--------------------------------------------------------------------------
Pair 3 Irrelevant Irrelevant System B (Retrieves 3)
Human/LLM Score: Low
As observed in the Report on the 1st Workshop on Large Language Model for Evaluation in Information Retrieval (LLM4Eval 2024) at SIGIR 2024, while LLMs exhibit variance when labeling individual documents, their errors are often uniform across different retrieval systems. Consequently, when aggregating these scores to determine which search engine is better, the relative ranking of the systems remains highly consistent with human-generated ground truth.
To optimize this process, researchers use hybrid frameworks. The authors of LLM-Assisted Relevance Assessments: When Should We Ask LLMs for Help? (sigir 2025) introduce an active learning approach that routes ambiguous, borderline cases to human assessors while allowing calibrated LLMs to handle clear-cut relevant or irrelevant documents.
Navigating Model Bias
While automated evaluation solves the scalability bottleneck, it introduces architectural bias. As Varsha Ramineni demonstrates in Towards Understanding Bias in Synthetic Data for Evaluation (arxiv 2025), LLM judges act as lenient assessors that systematically favor information retrieval systems built on similar neural architectures.
To mitigate this, modern frameworks focus on fine-tuning smaller, open-source models using highly specific, query-dependent guidelines distilled from larger frontier models, balancing cost, speed, and neutrality.
Key Foundational Work
A Machine Learning based Approach to Evaluating Retrieval Systems (2006) — contributed 1 concept elements to this direction.
Automatic construction of known-item finding test beds (2006) — contributed 1 concept elements to this direction.