TL;DR
While reasoning models optimized with verifiable rewards (RLVR) excel at math and coding, they perform significantly worse at capturing human annotator disagreement on subjective tasks. In contrast, naive Chain-of-Thought (CoT) improves standard RLHF models' ability to predict these disagreement distributions. This highlights a critical risk in blindly replacing human annotators with "smarter" reasoning LLMs on tasks where subjective variance matters.
What Problem Does This Paper Address?
Human annotators frequently disagree when labeling subjective text, such as identifying hate speech, classifying nuanced emotions, or rating chatbot preferences. Traditionally, Natural Language Processing (NLP) treats this disagreement as noise to be scrubbed away via majority voting. However, this variance often reflects genuine task ambiguity and a healthy diversity of human values.
As Large Language Models (LLMs) are increasingly used to replace human annotators or act as judges, they need to understand the full spectrum of human disagreement rather than just predicting a single majority label. This paper tackles a pressing question: Does the recent wave of "reasoning" LLMs—specifically those trained via Reinforcement Learning with Verifiable Rewards (RLVR), like the DeepSeek-R1 series—help or hurt when it comes to modeling human annotator disagreement?
Methods
The authors introduce a systematic evaluation framework to isolate the causal effect of reasoning on disagreement modeling. They map out 60 distinct experimental setups across combinations of:
- Three reasoning settings: Standard RLHF models without CoT, RLHF models with CoT, and RLVR models (DeepSeek-R1 series).
- Five model sizes: Ranging from 8B to 671B parameters to observe scaling effects.
- Two distribution expression methods: Verbalized probability (directly asking the LLM to state the percentage of humans who would pick a label) vs. Sampling-based distribution (generating 10 independent samples from the model and counting the frequencies).
- Steering methods: Evaluating the impact of zero-shot prompting against balanced few-shot prompting.
The performance is measured using two primary metrics: Variance Correlation (VarCorr), which checks if model uncertainty correlates with human uncertainty, and Distributional Alignment (DistAlign), which measures the exact divergence between the predicted and true human label distributions. They also compare these prompt-based LLM approaches against smaller encoder models (ModernBERT and DeBERTa-V3) fine-tuned directly on human distribution data.
Datasets
The evaluation spans five datasets across three widely studied subjective tasks:
- Gab Hate Corpus (GHC): Social media posts annotated for hate speech.
- GoEmotions: Fine-grained emotion classification, split into three binary subtasks: Positive (Pos), Negative (Neg), and Ambiguous (Amb).
- HelpSteer2 (HS2): Human preferences regarding chatbot response helpfulness.
For each dataset, the authors evaluate two distinct subsets: a Random subset (1k samples) representing the natural data distribution, and a HighVar subset (200 samples) specifically filtered to contain instances where human annotators strongly disagreed.
Key Insights and Ideas
- The "Objective Truth" Bias of RLVR: RLVR models are heavily optimized to find single, verifiable, deterministic answers (like a correct line of code or a math proof). Qualitative analysis reveals that this training backfires on subjective tasks: RLVR models strictly fixate on the literal annotation guidelines and assume all humans will interpret them identically, causing them to severely underestimate human disagreement.
- CoT Helps RLHF, But Not RLVR: Standard RLHF models benefit from thinking step-by-step, using CoT to explore multiple valid perspectives before outputting a distribution. However, forcing additional reasoning steps ("forcing longer thinking") in RLVR models fails to recover their performance drop.
- Just Ask for the Distribution: Directly prompting an LLM to output a verbalized probability distribution significantly outperforms the common practice of sampling multiple outputs to build an empirical distribution.
Key Results
- Verbalizing Wins Over Sampling: As shown in the win-rate matrix (Table 1), asking for verbalized probabilities achieves over a 92% win rate against sampling-based methods for predicting distributions, though sampling remains slightly better for predicting the raw majority label.
- RLVR Degrades Disagreement Modeling: On the high-disagreement HighVar subsets, RLVR models perform poorly, losing to their RLHF counterparts 82% of the time on distributional alignment (DistAlign).
- Human Labels Remain King: Smaller encoder models fine-tuned on actual human distributions consistently outperform even the largest zero-shot or few-shot LLMs on most tasks, proving that real human annotation data cannot be easily engineered away with prompts.
Limitations and Future Work
While the paper clearly demonstrates that RLVR harms disagreement modeling, the authors note that their qualitative analysis cannot fully explain some of the more erratic behaviors observed. For instance, why few-shot steering drastically improves performance on some datasets (like GHC) while completely tanking it on others (like HelpSteer2) remains an open question. Additionally, the study focuses entirely on binary classification setups. Future work will need to explore how these dynamics play out in multi-class settings and investigate more token-efficient ways to distill human disagreement into language models.
Summary updated