Human Feedback is not Gold Standard

Tom Hosking, Phil Blunsom, Max Bartolo

Abstract

Reinforcement Learning from Human Feedback (RLHF) has become the primary mechanism for aligning Large Language Models (LLMs) with human values. However, this study demonstrates that human preference is a flawed "gold standard." By deconstructing human judgment through error taxonomy and regression analysis, we show that evaluators are systematically biased toward model assertiveness and against refusals, often at the expense of factuality. Our results indicate that humans are "blinded" by confident language, leading to a significant gap in error detection between crowdworkers and experts. This suggests that current alignment techniques may be rewarding superficial traits over substantive accuracy.

Introduction

The AI community largely relies on human preference scores to train and evaluate LLMs. The assumption is that if a human prefers a response, it is inherently of higher quality. However, a single preference score is a "black box" that may obscure problematic biases. There is growing concern that human evaluators prioritize superficial characteristics—such as fluency, length, and tone—over harder-to-verify traits like factuality. This paper investigates whether human feedback is a reliable proxy for truth or if it encourages models to become "sophisticated sycophants" that prioritize sounding smart over being right.

Methods

The researchers employed a multi-stage methodology to analyze the components of human judgment:

  1. Error Taxonomy: Ten task-independent error types (e.g., Factuality, Inconsistency, Refusal, Repetition) were established.
  2. Lasso Regression: A regression model was used to determine the relative weighting of each error type in contributing to an overall human "quality" rating.
  3. The Preamble Technique: System prompts (preambles) were used to manipulate model outputs for assertiveness and linguistic complexity while keeping underlying facts constant.
  4. Crowd vs. Expert Comparison: Annotations from general crowdworkers were compared against "expert" annotations (the authors) to identify where non-experts fail to detect critical errors.

The study utilized three datasets—Curation Corpus (summarization), Amazon Product Descriptions, and Wikihow—and evaluated several models, including MPT 30B, Falcon 40B, Command 6B/52B, and Llama 2 13B Chat.

Results

The Dominance of the Refusal Penalty

The strongest driver of low human scores was found to be a refusal. Humans penalize models heavily when they state "I can't help with that," even for unreasonable requests. In the Lasso regression model, Refusal received a weighting of 0.95, whereas Factuality only scored 0.35. This suggests that appearing helpful is nearly three times more important to human scorers than being accurate.

Figure 1: Regression weighting of various error types on human quality scores. Refusal is the most significant negative driver.

The "Confidence Trick" and Assertiveness Bias

Assertiveness acts as a major confounder in human evaluation. When models use authoritative language, humans are significantly more likely to overlook factual errors. Assertiveness showed a strong positive correlation with overall quality scores (Pearson coefficient of 0.68), even when information content was identical.

The Detection Gap

Crowdworkers significantly underestimated factuality errors compared to experts. This gap widened in "high assertiveness" scenarios, where the difference in factuality error detection reached -22.3%. Humans struggle to disentangle different aspects of quality; if a response is fluent and relevant, they subconsciously assume it is also factual.

Figure 5: Heatmaps showing the change in error detection rates as a function of assertiveness (left) and complexity (right). Note the significant drop in factuality detection for high-assertiveness outputs.

Discussion and Conclusion

The findings suggest that RLHF may act as an "assertiveness booster." Preliminary evidence shows that RLHF-trained models (like Llama 2) are more assertive than those fine-tuned on preference scores alone for the same quality level. This creates a training pressure for models to hallucinate rather than admit ignorance.

Future Directions:

  • Transitioning toward reference-free evaluation to reduce reliance on subjective human "vibes."
  • Training models to be "humble," maintaining quality without artificial assertiveness.
  • Implementing "jury learning" or specialized annotator pools to mitigate individual biases.

While human feedback is essential for alignment, it is not a gold standard. We must develop more rigorous evaluation frameworks that can see past the "confidence trick."

Summary updated

BibTeX
@inproceedings{hosking2024human,
  author = {Tom Hosking and Phil Blunsom and Max Bartolo},
  title = {{Human Feedback is not Gold Standard}},
  booktitle = {International Conference on Learning Representations},
  year = {2024},
  url = {https://openreview.net/forum?id=7W3GLNImfS},
}
Figure