LLM-based relevance assessment still can't replace human relevance assessment

Charles L. A. Clarke, Laura Dietz

TL;DR

This paper argues that Large Language Models (LLMs) cannot yet replace human judgment for relevance assessment in information retrieval, despite recent claims suggesting otherwise. The authors highlight practical issues like the ease of gaming LLM-based evaluations and theoretical concerns such as LLM "narcissism" and the risk of circular evaluation loops.

What Problem Does This Paper Address?

The field of Information Retrieval (IR) relies heavily on relevance assessments to evaluate the performance of search systems. Traditionally, human judges meticulously assess whether documents retrieved by a search system are relevant to a given query. Recently, there's been a surge of interest in using LLMs to automate this process, with some studies suggesting LLM-based assessments are comparable to human ones. A recent paper by Upadhyay et al. [11] even claims that LLM-based assessments, specifically using the UMBRELA system, can fully replace human judgments in TREC-style evaluations. This paper critically examines that claim, arguing that this replacement is premature due to practical and theoretical limitations.

Methods

The authors critically analyze the findings of Upadhyay et al. [11] by:

  • Re-examining correlation metrics: They investigate the reported correlation between LLM and human judgments, specifically looking at how this correlation changes when focusing on top-performing systems.
  • Empirical Demonstration of Subversion: They describe a deliberately crafted submission (run uwc1) designed to exploit automatic evaluation metrics, demonstrating how easily LLM-based evaluations can be manipulated to achieve artificially high scores without genuine improvement in retrieval quality.
  • Simulation of Circularity: They simulate a scenario where LLM-based assessments are used both for evaluation and as a re-ranking mechanism within the systems being evaluated. This explores the potential consequences of a feedback loop where LLMs judge outputs generated by LLMs.

Datasets

The paper primarily uses data from the TREC 2024 RAG track. This track involved participants executing 301 queries over the MS MARCO Segment V2.1 collection, producing ranked lists of 100 passages per query. The authors also refer to data from the TREC 2021 Deep Learning track and the MS MARCO collection.

Key Insights and Ideas

  • Correlation weakens at the top: While LLM and human judgments show high correlation overall, this alignment significantly degrades when focusing on the top-performing systems, making it difficult to discern genuine progress.
  • Automatic evaluations are easily gamed: Systems can be designed to exploit the known mechanisms of LLM-based evaluators, leading to inflated scores that don't reflect true retrieval quality.
  • LLM "narcissism" is a real concern: LLMs tend to favor outputs generated by other LLMs, creating a bias that undermines objective evaluation.
  • Circularity is a looming threat: When LLMs are used for both evaluation and system components (like re-ranking), it creates a feedback loop where the evaluation metric becomes part of the system, rendering the evaluation invalid.
  • LLMs are not people: The fundamental purpose of IR systems is to serve human needs, and therefore, their evaluation must ultimately reflect human judgment and preferences, which LLMs currently cannot fully replicate.

Key Results

  • The correlation (Kendall's τ) between manual and UMBRELA assessments drops significantly for top-performing systems: from 0.85 for the top 60 systems to 0.51 for the top 20 systems.
  • A deliberately manipulated run (uwc1) ranked 5th under automatic assessment but only 28th under manual assessment, demonstrating the ease of subversion.
  • In a simulated circular evaluation scenario, the correlation (Kendall's τ) dropped sharply to 0.63 for the top 60 systems, and even became negative (-0.40) for the top 5 systems, indicating substantial score inflation.

Limitations and Future Work

Limitations: * The authors acknowledge that the TREC 2024 RAG track data used for some analyses was preliminary and that a full public data release was not available at the time of writing. * The simulation of circularity is a hypothetical scenario, though the authors argue it's a plausible future outcome.

Future Work: The paper strongly advocates for continued reliance on human judgments for relevance assessment. It implies that before LLMs can be considered viable replacements, significant theoretical challenges, such as LLM narcissism and the risk of circular evaluation, must be addressed. Future work should focus on developing safeguards against feedback loops and ensuring that LLM-based evaluations genuinely reflect human needs and preferences.

Summary updated

Abstract

The use of large language models (LLMs) for relevance assessment in information retrieval has gained significant attention, with recent studies suggesting that LLM-based judgments provide comparable evaluations to human judgments. Notably, based on TREC 2024 data, Upadhyay et al. make a bold claim that LLM-based relevance assessments, such as those generated by the UMBRELA system, can fully replace traditional human relevance assessments in TREC-style evaluations. This paper critically examines this claim, highlighting practical and theoretical limitations that undermine the validity of this conclusion. First, we question whether the evidence provided by Upadhyay et al. really supports their claim, particularly if a test collection is used asa benchmark for future improvements. Second, through a submission deliberately intended to do so, we demonstrate the ease with which automatic evaluation metrics can be subverted, showing that systems designed to exploit these evaluations can achieve artificially high scores. Theoretical challenges -- such as the inherent narcissism of LLMs, the risk of overfitting to LLM-based metrics, and the potential degradation of future LLM performance -- must be addressed before LLM-based relevance assessments can be considered a viable replacement for human judgments.
BibTeX
@inproceedings{clarke2025llm,
  author = {Charles L. A. Clarke and Laura Dietz},
  title = {{LLM-based relevance assessment still can't replace human relevance assessment}},
  booktitle = {EVIA},
  year = {2025},
  url = {https://openreview.net/forum?id=qrGCuIbVAb},
}
Figure