Human Feedback is not Gold Standard

Tom Hosking, Phil Blunsom, Max Bartolo

Introduction

Reinforcement Learning from Human Feedback (RLHF) is the primary engine behind state-of-the-art Large Language Models (LLMs), credited with making models like ChatGPT feel helpful and conversational. In this paradigm, models are evaluated and trained using human preference scores—essentially asking annotators, "Which of these two answers do you like better?"

However, it remains unclear what a "preference score" truly represents. Is a human choosing an answer because it is factually correct, or simply because it is fluent and sounds authoritative? If human annotators are biased toward certain writing styles, we might inadvertently train models to prioritize "vibes" over veracity. This paper investigates whether human feedback accurately captures critical error criteria, such as factuality and consistency, or if it is heavily swayed by confounders like linguistic complexity and assertiveness.

Methodology

To deconstruct human judgment, the researchers conducted a multi-stage study:

  1. Error Taxonomy: They established 10 task-independent error types (e.g., Factuality, Inconsistency, Redundancy, Refusal) to determine how each contributes to an overall quality score.
  2. Regression Analysis: By asking one group of annotators to mark specific errors and another to give an overall 1-5 quality score, they used a Lasso regression model to calculate the "weight" of each error type in the final human preference.
  3. Controlled Confounding: To test for bias, they used system prompts (preambles) to force models to generate outputs with varying levels of assertiveness (e.g., "Respond authoritatively...") and complexity (e.g., "Use technical terms...").
  4. Expert vs. Crowd Comparison: They compared annotations from general crowdworkers against "expert" annotations (from the authors) to identify where non-experts miss critical errors.

The study utilized three diverse datasets (Curation Corpus, Amazon Product Descriptions, and Wikihow) and tested several high-performing models, including Command, Llama 2 13B Chat, Falcon 40B, and MPT 30B.

Key Findings

The Weight of Errors: Refusal Over Truth

The regression analysis revealed a surprising hierarchy in what drives human preference. The single strongest predictor of a low human preference score was not a hallucination or a factual error, but a refusal to answer.

Regression weights of different error types on overall human preference scores.

As shown in the regression weighting, Refusal had the highest negative weight (0.95). In contrast, Factuality (0.35) and Inconsistency (0.21) were weighted much lower. This suggests that humans penalize a model saying "I can't help with that" much more harshly than they penalize a model providing a fluent but factually incorrect response. "Helpfulness" effectively outweighs "truth" in general human preference.

The "Confidence Trap"

Assertiveness proved to be a massive confounder in human evaluation. When a model speaks confidently, humans are significantly less likely to notice factual errors or inconsistencies.

Change in perceived error rates based on the assertiveness of the model's output.

By artificially increasing the assertiveness of the generated text, the perceived rate of factuality errors dropped dramatically. The "expert" check revealed that crowdworkers consistently underestimated factuality errors in these assertive outputs, widening the gap between expert and non-expert detection. Annotators are simply more trusting of assertive responses, falling into a "confidence trap."

Additional Insights

  • Disentanglement Failure: Humans struggle to separate different aspects of quality. If a response is irrelevant to the prompt, annotators tend to rate it as "less factual" even if the statements within it are technically true.
  • Length Bias: Annotators generally prefer longer responses until the model begins to repeat itself, confirming that verbosity is often mistaken for quality.
  • RLHF Side Effects: Preliminary evidence suggests that RLHF might train models to be more assertive as a shortcut to higher preference scores. Models trained with RLHF showed higher assertiveness for equivalent quality levels compared to models trained on simpler preference fine-tuning.

Limitations and Future Directions

The authors acknowledge that their "expert" dataset is relatively small (300 examples) and that the study focused on specific tasks like summarization and product descriptions. The dynamics of human feedback might shift in more technical domains like coding or mathematics.

Ultimately, the paper serves as a critical call to action: the AI community needs to move beyond "single-score" human feedback. Future work must focus on developing robust evaluation metrics that cannot be gamed by assertive or verbose styles, ensuring that the models of tomorrow are optimized for correctness, not just confidence.

Summary updated

Abstract

Human feedback has become the de facto standard for evaluating the performance of Large Language Models, and is increasingly being used as a training objective. However, it is not clear which properties of a generated output this single `preference' score captures. We hypothesise that preference scores are subjective and open to undesirable biases. We critically analyse the use of human feedback for both training and evaluation, to verify whether it fully captures a range of crucial error criteria. We find that while preference scores have fairly good coverage, they under-represent important aspects like factuality. We further hypothesise that both preference scores and error annotation may be affected by confounders, and leverage instruction-tuned models to generate outputs that vary along two possible confounding dimensions: assertiveness and complexity. We find that the assertiveness of an output skews the perceived rate of factuality errors, indicating that human annotations are not a fully reliable evaluation metric or training objective. Finally, we offer preliminary evidence that using human feedback as a training objective disproportionately increases the assertiveness of model outputs. We encourage future work to carefully consider whether preference scores are well aligned with the desired objective.
BibTeX
@inproceedings{hosking2024human,
  author = {Tom Hosking and Phil Blunsom and Max Bartolo},
  title = {{Human Feedback is not Gold Standard}},
  booktitle = {International Conference on Learning Representations},
  year = {2024},
  url = {https://openreview.net/forum?id=7W3GLNImfS},
}
Figure