LLM-Evaluation Tropes: Perspectives on the Validity of LLM-Evaluations

Laura Dietz, Oleg Zendel, Peter Bailey, Charles L. A. Clarke, Ellese Cotterill, Jeff Dalton, Faegheh Hasibi, Mark Sanderson, Nick CraswellarXiv 2025 · DOI · arXiv · OpenReview

TL;DR

This paper identifies and categorizes "tropes" – common pitfalls and biases – that can undermine the validity of using Large Language Models (LLMs) for evaluating Information Retrieval (IR) systems. It proposes guardrails and a collaborative framework to ensure LLM-based evaluations remain scientifically rigorous and trustworthy, preventing misleading conclusions and ensuring systems truly benefit users.

What Problem Does This Paper Address?

Traditionally, human assessors have been the gold standard for judging the relevance of documents to a user's query in Information Retrieval (IR). This process is crucial for evaluating and improving search systems. However, the sheer scale of data and the complexity of modern IR systems (like generative models) make manual evaluation increasingly impractical and expensive. Large Language Models (LLMs) offer a promising alternative, capable of generating relevance judgments much faster and at a lower cost. While initial studies suggest LLMs can align with human judgments, there's a growing concern that their unchecked use could lead to flawed evaluations. This paper addresses the critical problem of ensuring the validity of LLM-based evaluations, preventing situations where LLMs might falsely indicate success, leading to the development and deployment of systems that don't actually perform well in real-world scenarios. The core challenge is to harness the scalability of LLMs without sacrificing the scientific integrity and trustworthiness of IR evaluation.

Methods

The paper's primary contribution is a comprehensive taxonomy of "LLM-Evaluation Tropes" – recurring patterns that can negatively impact evaluation validity. These tropes are categorized into four groups: Evaluation Tropes (affecting measurements), Meta-Evaluation Tropes (affecting perceived quality), System Tropes (degrading system performance), and Judge Tropes (undermining human judgment). For each trope, the authors explain the mechanism, provide examples, and suggest methods to quantify its effect and implement guardrails to mitigate it. They also propose a "Coopetition" framework for building reusable, collaborative test collections that integrate LLM judgments responsibly.

Datasets

The paper discusses the use of existing datasets and benchmarks, particularly mentioning the TREC RAG 2024 track data for demonstrating the "Circularity" trope. They also reference the use of UMBRELA (an open-source toolkit for LLM-based relevance assessment) and its application in a TREC task. The proposed "Coopetition" framework aims to create new, collaborative test collections that are regularly updated with recent systems and human judgments.

Key Insights and Ideas

  • LLMs aren't magic bullets for evaluation: While LLMs offer speed and scale, they introduce new, subtle biases and risks that can lead to misleading results if not carefully managed.
  • Circularity is a major threat: When LLM-generated evaluation signals influence the development of the very systems being evaluated, it creates a feedback loop that inflates performance metrics without real-world improvement. This is akin to "training for the test."
  • LLM "Narcissism" is real: LLMs can exhibit a bias towards text that aligns with their own generation patterns, potentially favoring outputs from similar models over genuinely better, but stylistically different, ones.
  • Human oversight is still crucial, but tricky: While human judges are essential, the "Rubber-Stamp Effect" shows that humans can blindly trust LLM outputs, negating the benefit of their involvement.
  • Reproducibility is challenged by LLM evolution: LLMs are constantly updated, making it difficult to reproduce past evaluations if the specific model version used is no longer available or has changed its behavior.
  • A collaborative approach is needed: The proposed "Coopetition" model emphasizes building shared, evolving test collections and evaluation methodologies to foster trust and address the dynamic nature of LLM development.

Key Results

The paper doesn't present traditional quantitative results from a single experiment but rather a qualitative analysis and categorization of risks. However, they demonstrate the "Circularity" trope using TREC RAG 2024 data. Their analysis shows that when systems are re-ranked using the UMBRELA LLM evaluator and then evaluated with the same LLM's relevance labels, there's a significant increase in disagreement between LLM and human judgments (higher discordance, lower Kendall's tau). Specifically, they observed a decrease in Kendall's tau from 0.84 (original systems vs. manual) to 0.63 (re-ranked systems vs. manual) for the top 60 systems. They also highlight cases where LLM evaluations showed near-perfect scores (NDCG > 0.95) while manual evaluations were much lower (0.68-0.72), illustrating the inflated performance due to circularity.

Limitations and Future Work

The authors acknowledge that LLMs are not static and can evolve, posing challenges for long-term evaluation validity. They also note that even with guardrails, some biases might persist due to overlapping training data. Future work includes:

  • Developing robust guardrails: Continuously refining methods to quantify and mitigate the identified tropes.
  • Building reusable test collections: Establishing a collaborative framework (like the proposed "Coopetition") for creating and maintaining up-to-date test collections.
  • Standardizing LLM evaluation practices: Promoting best practices to ensure LLM-based evaluations are trustworthy, reproducible, and scientifically sound.
  • Investigating adversarial threats: Developing methods to test the resilience of LLM evaluators and IR systems against manipulation.

Summary updated

Abstract

Large Language Models (LLMs) are increasingly used to evaluate information retrieval (IR) systems, generating relevance judgments traditionally made by human assessors. Recent empirical studies suggest that LLM-based evaluations often align with human judgments, leading some to suggest that human judges may no longer be necessary, while others highlight concerns about judgment reliability, validity, and long-term impact. As IR systems begin incorporating LLM-generated signals, evaluation outcomes risk becoming self-reinforcing, potentially leading to misleading conclusions. This paper examines scenarios where LLM-evaluators may falsely indicate success, particularly when LLM-based judgments influence both system development and evaluation. We highlight key risks, including bias reinforcement, reproducibility challenges, and inconsistencies in assessment methodologies. To address these concerns, we propose tests to quantify adverse effects, guardrails, and a collaborative framework for constructing reusable test collections that integrate LLM judgments responsibly. By providing perspectives from academia and industry, this work aims to establish best practices for the principled use of LLMs in IR evaluation.
BibTeX
@misc{dietz2025llm,
  author = {Laura Dietz and Oleg Zendel and Peter Bailey and Charles L. A. Clarke and Ellese Cotterill and Jeff Dalton and Faegheh Hasibi and Mark Sanderson and Nick Craswell},
  title = {{LLM-Evaluation Tropes: Perspectives on the Validity of LLM-Evaluations}},
  year = {2025},
  eprint = {2504.19076},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2504.19076},
}
Figure