A Human-AI Comparative Analysis of Prompt Sensitivity in LLM-Based Relevance Judgment

Negar Arabzadeh, Charles L. A. Clarke

TL;DR

This paper investigates how sensitive Large Language Models (LLMs) are to different prompts when performing relevance judgments for information retrieval tasks. The authors found that while LLMs can achieve high agreement with human judgments, their performance varies significantly depending on the prompt's wording and the LLM used as a judge, especially for graded relevance tasks. This work highlights the need for careful prompt engineering and suggests LLM-generated prompts might offer more consistent results.

What Problem Does This Paper Address?

Information retrieval (IR) systems are increasingly being evaluated using Large Language Models (LLMs) for relevance judgments, with LLM performance often approaching inter-human agreement. However, the effectiveness of LLMs in this role is heavily influenced by the prompts used to guide them. Previous studies have often used prompts tailored to specific LLMs or datasets. This paper addresses the gap by systematically analyzing the impact of prompt sensitivity on LLM-based relevance judgments across different LLMs and relevance assessment approaches (binary, graded, and pairwise). The goal is to understand how robust LLM judgments are to prompt variations and to compare the effectiveness of human-generated versus LLM-generated prompts.

Methods

The researchers collected prompts for relevance assessment from two sources: 15 human experts and 15 different LLMs. These prompts were designed to cover three relevance judgment tasks: binary (relevant/not relevant), graded (0-3 scale), and pairwise (comparing two passages). After filtering, 72 prompts were used to evaluate document-query pairs from TREC Deep Learning datasets. Three LLMs (GPT-40, LLAMA 3.2, and Mistral-7B) were used as judges. Agreement with human annotations was measured using Cohen's Kappa for binary and graded judgments, and pairwise agreement for pairwise judgments. They also analyzed prompt diversity and compared LLM performance with a standardized prompt (UMBRELA).

Datasets

  • TREC Deep Learning Track datasets:
    • DL 2020 (MS MARCO V1 collection)
    • DL 2021 (MS MARCO V2 collection)

These datasets contain manually annotated relevance judgments by NIST assessors. The paper also makes its collected prompts and experimental data publicly available.

Key Insights and Ideas

  • Prompt Sensitivity Varies by Task: Graded relevance judgments are significantly more sensitive to prompt variations than binary or pairwise judgments. This suggests that the nuanced nature of graded scales requires more precise prompting.
  • LLM-Generated Prompts Offer Consistency: Prompts generated by LLMs tend to show lower variance in agreement compared to human-crafted prompts. This indicates they might be more robust to minor prompt modifications.
  • Human Prompts Show Greater Diversity: Human-generated prompts exhibit more lexical diversity, suggesting humans introduce more nuanced phrasing. This can be a double-edged sword, leading to richer prompts but potentially less consistent LLM responses.
  • UMBRELA as a Strong Baseline: The UMBRELA prompt, a reproduction of Bing's LLM assessor, consistently performed well across different LLMs and tasks, particularly for graded relevance. Its structured approach, which breaks down relevance into finer aspects, is hypothesized to be key to its robustness.
  • LLM Judges Differ in Sensitivity: Not all LLMs are equally sensitive to prompt changes. GPT-40 showed high consistency across prompts, while LLAMA 3.2 and Mistral exhibited more variability.

Key Results

  • Agreement with Human Judgments: LLM-generated prompts generally showed higher average agreement with human annotations, though the difference was minimal for graded relevance.
  • Variance in Agreement: LLM-generated prompts exhibited lower variance in agreement compared to human-crafted prompts.
  • UMBRELA Performance: UMBRELA demonstrated high and consistent performance across different LLMs and tasks, especially for graded relevance.
  • Model-Specific Performance: GPT-40 showed high consistency in agreement, while LLAMA 3.2 and Mistral showed significant fluctuations depending on the prompt and assessment method.

Limitations and Future Work

  • Limited LLM Judges: The study only evaluated three LLMs as judges, limiting the generalizability of findings regarding LLM-specific sensitivity.
  • Human Participant Background: Human participants primarily had computer science backgrounds and experience with LLMs, which might influence the nature of the prompts they generated.

Future work could involve evaluating a wider range of LLMs and human annotators with diverse backgrounds. Further investigation into the specific linguistic features of prompts that lead to better LLM performance is also suggested. The authors also emphasize the importance of prompt engineering for reliable LLM-based evaluation.

Summary updated

Abstract

Large Language Models (LLMs) are increasingly used to automate relevance judgments for information retrieval (IR) tasks, often demonstrating agreement with human labels that approaches inter-human agreement. To assess the robustness and reliability of LLM-based relevance judgments, we systematically investigate impact of prompt sensitivity on the task. We collected prompts for relevance assessment from 15 human experts and 15 LLMs across three tasks~ -- ~binary, graded, and pairwise~ -- ~yielding 90 prompts in total. After filtering out unusable prompts from three humans and three LLMs, we employed the remaining 72 prompts with three different LLMs as judges to label document/query pairs from two TREC Deep Learning Datasets (2020 and 2021). We compare LLM-generated labels with TREC official human labels using Cohen's $κ$ and pairwise agreement measures. In addition to investigating the impact of prompt variations on agreement with human labels, we compare human- and LLM-generated prompts and analyze differences among different LLMs as judges. We also compare human- and LLM-generated prompts with the standard UMBRELA prompt used for relevance assessment by Bing and TREC 2024 Retrieval Augmented Generation (RAG) Track. To support future research in LLM-based evaluation, we release all data and prompts at https://github.com/Narabzad/prompt-sensitivity-relevance-judgements/.
BibTeX
@inproceedings{arabzadeh2025human,
  author = {Negar Arabzadeh and Charles L. A. Clarke},
  title = {{A Human-AI Comparative Analysis of Prompt Sensitivity in LLM-Based Relevance Judgment}},
  booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
  year = {2025},
  url = {https://openreview.net/forum?id=3QMR3NqUGB},
}
Figure