TL;DR
This paper investigates how sensitive Large Language Models (LLMs) are to different prompts when performing relevance judgments for information retrieval tasks. The authors found that while LLMs can achieve high agreement with human judgments, their performance varies significantly depending on the prompt's wording and the LLM used as a judge, especially for graded relevance tasks. This work highlights the need for careful prompt engineering and suggests LLM-generated prompts might offer more consistent results.
What Problem Does This Paper Address?
Information retrieval (IR) systems are increasingly being evaluated using Large Language Models (LLMs) for relevance judgments, with LLM performance often approaching inter-human agreement. However, the effectiveness of LLMs in this role is heavily influenced by the prompts used to guide them. Previous studies have often used prompts tailored to specific LLMs or datasets. This paper addresses the gap by systematically analyzing the impact of prompt sensitivity on LLM-based relevance judgments across different LLMs and relevance assessment approaches (binary, graded, and pairwise). The goal is to understand how robust LLM judgments are to prompt variations and to compare the effectiveness of human-generated versus LLM-generated prompts.
Methods
The researchers collected prompts for relevance assessment from two sources: 15 human experts and 15 different LLMs. These prompts were designed to cover three relevance judgment tasks: binary (relevant/not relevant), graded (0-3 scale), and pairwise (comparing two passages). After filtering, 72 prompts were used to evaluate document-query pairs from TREC Deep Learning datasets. Three LLMs (GPT-40, LLAMA 3.2, and Mistral-7B) were used as judges. Agreement with human annotations was measured using Cohen's Kappa for binary and graded judgments, and pairwise agreement for pairwise judgments. They also analyzed prompt diversity and compared LLM performance with a standardized prompt (UMBRELA).
Datasets
- TREC Deep Learning Track datasets:
- DL 2020 (MS MARCO V1 collection)
- DL 2021 (MS MARCO V2 collection)
These datasets contain manually annotated relevance judgments by NIST assessors. The paper also makes its collected prompts and experimental data publicly available.
Key Insights and Ideas
- Prompt Sensitivity Varies by Task: Graded relevance judgments are significantly more sensitive to prompt variations than binary or pairwise judgments. This suggests that the nuanced nature of graded scales requires more precise prompting.
- LLM-Generated Prompts Offer Consistency: Prompts generated by LLMs tend to show lower variance in agreement compared to human-crafted prompts. This indicates they might be more robust to minor prompt modifications.
- Human Prompts Show Greater Diversity: Human-generated prompts exhibit more lexical diversity, suggesting humans introduce more nuanced phrasing. This can be a double-edged sword, leading to richer prompts but potentially less consistent LLM responses.
- UMBRELA as a Strong Baseline: The UMBRELA prompt, a reproduction of Bing's LLM assessor, consistently performed well across different LLMs and tasks, particularly for graded relevance. Its structured approach, which breaks down relevance into finer aspects, is hypothesized to be key to its robustness.
- LLM Judges Differ in Sensitivity: Not all LLMs are equally sensitive to prompt changes. GPT-40 showed high consistency across prompts, while LLAMA 3.2 and Mistral exhibited more variability.
Key Results
- Agreement with Human Judgments: LLM-generated prompts generally showed higher average agreement with human annotations, though the difference was minimal for graded relevance.
- Variance in Agreement: LLM-generated prompts exhibited lower variance in agreement compared to human-crafted prompts.
- UMBRELA Performance: UMBRELA demonstrated high and consistent performance across different LLMs and tasks, especially for graded relevance.
- Model-Specific Performance: GPT-40 showed high consistency in agreement, while LLAMA 3.2 and Mistral showed significant fluctuations depending on the prompt and assessment method.
Limitations and Future Work
- Limited LLM Judges: The study only evaluated three LLMs as judges, limiting the generalizability of findings regarding LLM-specific sensitivity.
- Human Participant Background: Human participants primarily had computer science backgrounds and experience with LLMs, which might influence the nature of the prompts they generated.
Future work could involve evaluating a wider range of LLMs and human annotators with diverse backgrounds. Further investigation into the specific linguistic features of prompts that lead to better LLM performance is also suggested. The authors also emphasize the importance of prompt engineering for reliable LLM-based evaluation.
Summary updated