On a Spurious Interaction between Uncertainty Scores and Answer Evaluation Metrics in Generative QA Tasks

Andrea Santilli, Miao Xiong, Michael Kirchhof, Pau Rodriguez, Federico Danieli, Xavier Suau, Luca Zappella, Sinead Williamson, Adam GolinskiSafeGenAi @ NeurIPS 2024 · OpenReview

TL;DR

This paper reveals a hidden bias in how we evaluate the uncertainty of Large Language Models (LLMs). It shows that common evaluation metrics, particularly those based on text overlap, can be spuriously correlated with response length, leading to misleading conclusions about the effectiveness of uncertainty quantification methods. The authors advocate for using LLM-as-a-judge evaluation as a more robust alternative.

What Problem Does This Paper Address?

As LLMs become more powerful and are deployed in critical applications, it's crucial to know when they are uncertain about their outputs. This field, known as Uncertainty Quantification (UQ), aims to develop methods that reliably signal an LLM's confidence. However, a significant challenge is that reported UQ performance across different research papers is often incomparable and even contradictory. This paper investigates why these discrepancies occur, focusing on the interaction between UQ methods and the metrics used to evaluate their performance, specifically in generative Question Answering (QA) tasks.

Methods

The core of this paper is an empirical investigation into the evaluation protocols used for UQ in LLMs. The authors compare two main approaches:

  • Substring-overlap-based evaluation: This uses metrics like Rouge-L or SQuAD to compare the generated answer to a reference answer based on shared text.
  • LLM-as-a-judge evaluation: This employs a separate, powerful LLM to assess whether the generated answer is semantically equivalent to the reference answer.

They specifically examine how the choice of evaluation metric affects the perceived performance of various UQ methods, particularly those sensitive to response length, like sequence probability. The paper also delves into the nuances of different UQ methods, categorizing them into single-sample, multiple-sample, and learned approaches.

Datasets

The study utilizes several standard QA datasets commonly found in UQ literature:

  • TriviaQA: Tests trivia facts.
  • SQuAD: General knowledge questions answerable from context.
  • NQ-Open: Natural Google queries.
  • SVAMP: Math word problems.

The authors note that they omit BioASQ due to restricted access. For their analysis, they focus on the long-form generation setting.

Key Insights and Ideas

  • The "Response Length" Confound: The most significant finding is that certain answer evaluation metrics (especially substring-overlap-based ones like Rouge-L) are spuriously correlated with the length of the LLM's generated response. This is problematic because UQ methods sensitive to response length (like sequence probability, which tends to decrease with longer sequences) can appear to perform better than they actually do.
  • Conflicting Conclusions: The paper demonstrates how this interaction explains the conflicting results seen in prior work. For instance, studies using Rouge-L might favor certain UQ methods, while those using LLM-as-a-judge might reach different conclusions, not due to fundamental differences in the UQ methods themselves, but due to the evaluation metric's bias.
  • LLM-as-a-Judge is More Robust: The authors provide evidence, aligning with previous research, that LLM-as-a-judge evaluation metrics exhibit higher agreement with human annotators compared to traditional substring-overlap metrics. This suggests they are a more reliable way to assess answer correctness.
  • Binarization Isn't a Silver Bullet: Even when substring-overlap scores are binarized (e.g., using a threshold), a correlation with response length persists, leading to a similar spurious effect on UQ performance.

Key Results

  • When using Rouge-L (both continuous and binarized), UQ methods like Sequence Probability and P(IK) appear to be top performers for many models.
  • When using an LLM-as-a-judge metric (specifically, Llama3-8b-instruct), the performance landscape shifts, and these methods no longer consistently rank as highly.
  • Human annotator agreement with Rouge-L is around 70%, while agreement with the LLM-as-a-judge metric exceeds 91%, highlighting the latter's superior reliability.
  • The paper visually demonstrates that for binarized Rouge-L, the proportion of False Negatives (answers deemed incorrect by the metric but correct by humans) increases with response length.

Limitations and Future Work

  • Response Length Bias: The primary limitation identified is the inherent bias of certain evaluation metrics towards response length. This can lead to misleading conclusions about UQ method performance.
  • LLM-as-a-Judge Nuances: While more robust, LLM-as-a-judge methods themselves have design choices (e.g., prompt engineering) that could influence results, though the paper focuses on the metric's inherent properties.
  • Future Work Recommendations: The authors suggest practitioners should favor LLM-as-a-judge metrics for more reliable evaluations. They also propose exploring better thresholds for binarizing scores to mitigate the length bias and investigating how learned UQ approaches might exploit this spurious interaction.

Summary updated

Abstract

Knowing when a language model is uncertain about its generations is a key challenge for enhancing LLMs’ safety and reliability. An increasing issue in the field of Uncertainty Quantification (UQ) for Large Language Models (LLMs) is that the performance values reported across papers are often incomparable, and sometimes even conflicting, due to different evaluation protocols. In this paper, we highlight that some UQ methods and answer evaluation metrics are spuriously correlated via the response length, which leads to falsely elevated performances of uncertainty scores that are sensitive to response length, such as sequence probability. We perform empirical evaluations according to two different protocols in the related literature, one using a substring-overlap-based evaluation metric, and one using an LLM-as-a-judge approach, and show that the conflicting conclusions between these two works can be attributed to this interaction.
BibTeX
@inproceedings{santilli2024spurious,
  author = {Andrea Santilli and Miao Xiong and Michael Kirchhof and Pau Rodriguez and Federico Danieli and Xavier Suau and Luca Zappella and Sinead Williamson and Adam Golinski},
  title = {{On a Spurious Interaction between Uncertainty Scores and Answer Evaluation Metrics in Generative QA Tasks}},
  booktitle = {SafeGenAi @ NeurIPS},
  year = {2024},
  url = {https://openreview.net/forum?id=jGtL0JFdeD},
}
Figure