Abstract
Reinforcement Learning from Human Feedback (RLHF) has become the primary mechanism for aligning Large Language Models (LLMs) with human values. However, this study demonstrates that human preference is a flawed "gold standard." By deconstructing human judgment through error taxonomy and regression analysis, we show that evaluators are systematically biased toward model assertiveness and against refusals, often at the expense of factuality. Our results indicate that humans are "blinded" by confident language, leading to a significant gap in error detection between crowdworkers and experts. This suggests that current alignment techniques may be rewarding superficial traits over substantive accuracy.
Introduction
The AI community largely relies on human preference scores to train and evaluate LLMs. The assumption is that if a human prefers a response, it is inherently of higher quality. However, a single preference score is a "black box" that may obscure problematic biases. There is growing concern that human evaluators prioritize superficial characteristics—such as fluency, length, and tone—over harder-to-verify traits like factuality. This paper investigates whether human feedback is a reliable proxy for truth or if it encourages models to become "sophisticated sycophants" that prioritize sounding smart over being right.
Methods
The researchers employed a multi-stage methodology to analyze the components of human judgment:
- Error Taxonomy: Ten task-independent error types (e.g., Factuality, Inconsistency, Refusal, Repetition) were established.
- Lasso Regression: A regression model was used to determine the relative weighting of each error type in contributing to an overall human "quality" rating.
- The Preamble Technique: System prompts (preambles) were used to manipulate model outputs for assertiveness and linguistic complexity while keeping underlying facts constant.
- Crowd vs. Expert Comparison: Annotations from general crowdworkers were compared against "expert" annotations (the authors) to identify where non-experts fail to detect critical errors.
The study utilized three datasets—Curation Corpus (summarization), Amazon Product Descriptions, and Wikihow—and evaluated several models, including MPT 30B, Falcon 40B, Command 6B/52B, and Llama 2 13B Chat.
Results
The Dominance of the Refusal Penalty
The strongest driver of low human scores was found to be a refusal. Humans penalize models heavily when they state "I can't help with that," even for unreasonable requests. In the Lasso regression model, Refusal received a weighting of 0.95, whereas Factuality only scored 0.35. This suggests that appearing helpful is nearly three times more important to human scorers than being accurate.
The "Confidence Trick" and Assertiveness Bias
Assertiveness acts as a major confounder in human evaluation. When models use authoritative language, humans are significantly more likely to overlook factual errors. Assertiveness showed a strong positive correlation with overall quality scores (Pearson coefficient of 0.68), even when information content was identical.
The Detection Gap
Crowdworkers significantly underestimated factuality errors compared to experts. This gap widened in "high assertiveness" scenarios, where the difference in factuality error detection reached -22.3%. Humans struggle to disentangle different aspects of quality; if a response is fluent and relevant, they subconsciously assume it is also factual.
Discussion and Conclusion
The findings suggest that RLHF may act as an "assertiveness booster." Preliminary evidence shows that RLHF-trained models (like Llama 2) are more assertive than those fine-tuned on preference scores alone for the same quality level. This creates a training pressure for models to hallucinate rather than admit ignorance.
Future Directions:
- Transitioning toward reference-free evaluation to reduce reliance on subjective human "vibes."
- Training models to be "humble," maintaining quality without artificial assertiveness.
- Implementing "jury learning" or specialized annotator pools to mitigate individual biases.
While human feedback is essential for alignment, it is not a gold standard. We must develop more rigorous evaluation frameworks that can see past the "confidence trick."
Summary updated