Can Reasoning Help Large Language Models Capture Human Annotator Disagreement?

Jingwei Ni, Yu Fan, Vilém Zouhar, Donya Rooein, Alexander Hoyle, Mrinmaya Sachan, Markus Leippold, Dirk Hovy, Elliott Ash

TL;DR

While reasoning models optimized with verifiable rewards (RLVR) excel at math and coding, they perform significantly worse at capturing human annotator disagreement on subjective tasks. In contrast, naive Chain-of-Thought (CoT) improves standard RLHF models' ability to predict these disagreement distributions. This highlights a critical risk in blindly replacing human annotators with "smarter" reasoning LLMs on tasks where subjective variance matters.

What Problem Does This Paper Address?

Human annotators frequently disagree when labeling subjective text, such as identifying hate speech, classifying nuanced emotions, or rating chatbot preferences. Traditionally, Natural Language Processing (NLP) treats this disagreement as noise to be scrubbed away via majority voting. However, this variance often reflects genuine task ambiguity and a healthy diversity of human values.

As Large Language Models (LLMs) are increasingly used to replace human annotators or act as judges, they need to understand the full spectrum of human disagreement rather than just predicting a single majority label. This paper tackles a pressing question: Does the recent wave of "reasoning" LLMs—specifically those trained via Reinforcement Learning with Verifiable Rewards (RLVR), like the DeepSeek-R1 series—help or hurt when it comes to modeling human annotator disagreement?

Methods

The authors introduce a systematic evaluation framework to isolate the causal effect of reasoning on disagreement modeling. They map out 60 distinct experimental setups across combinations of:

  • Three reasoning settings: Standard RLHF models without CoT, RLHF models with CoT, and RLVR models (DeepSeek-R1 series).
  • Five model sizes: Ranging from 8B to 671B parameters to observe scaling effects.
  • Two distribution expression methods: Verbalized probability (directly asking the LLM to state the percentage of humans who would pick a label) vs. Sampling-based distribution (generating 10 independent samples from the model and counting the frequencies).
  • Steering methods: Evaluating the impact of zero-shot prompting against balanced few-shot prompting.

The performance is measured using two primary metrics: Variance Correlation (VarCorr), which checks if model uncertainty correlates with human uncertainty, and Distributional Alignment (DistAlign), which measures the exact divergence between the predicted and true human label distributions. They also compare these prompt-based LLM approaches against smaller encoder models (ModernBERT and DeBERTa-V3) fine-tuned directly on human distribution data.

Figure 1: An illustration of our evaluation: We start with a task with guidelines for both human and LLM annotators. The LLM predictions of the annotation distributions are then compared with true human label distribution.

Datasets

The evaluation spans five datasets across three widely studied subjective tasks:

  1. Gab Hate Corpus (GHC): Social media posts annotated for hate speech.
  2. GoEmotions: Fine-grained emotion classification, split into three binary subtasks: Positive (Pos), Negative (Neg), and Ambiguous (Amb).
  3. HelpSteer2 (HS2): Human preferences regarding chatbot response helpfulness.

For each dataset, the authors evaluate two distinct subsets: a Random subset (1k samples) representing the natural data distribution, and a HighVar subset (200 samples) specifically filtered to contain instances where human annotators strongly disagreed.

Key Insights and Ideas

  • The "Objective Truth" Bias of RLVR: RLVR models are heavily optimized to find single, verifiable, deterministic answers (like a correct line of code or a math proof). Qualitative analysis reveals that this training backfires on subjective tasks: RLVR models strictly fixate on the literal annotation guidelines and assume all humans will interpret them identically, causing them to severely underestimate human disagreement.
  • CoT Helps RLHF, But Not RLVR: Standard RLHF models benefit from thinking step-by-step, using CoT to explore multiple valid perspectives before outputting a distribution. However, forcing additional reasoning steps ("forcing longer thinking") in RLVR models fails to recover their performance drop.
  • Just Ask for the Distribution: Directly prompting an LLM to output a verbalized probability distribution significantly outperforms the common practice of sampling multiple outputs to build an empirical distribution.

Key Results

  • Verbalizing Wins Over Sampling: As shown in the win-rate matrix (Table 1), asking for verbalized probabilities achieves over a 92% win rate against sampling-based methods for predicting distributions, though sampling remains slightly better for predicting the raw majority label.
  • RLVR Degrades Disagreement Modeling: On the high-disagreement HighVar subsets, RLVR models perform poorly, losing to their RLHF counterparts 82% of the time on distributional alignment (DistAlign).
  • Human Labels Remain King: Smaller encoder models fine-tuned on actual human distributions consistently outperform even the largest zero-shot or few-shot LLMs on most tasks, proving that real human annotation data cannot be easily engineered away with prompts.
Win rates of different experimental settings across metrics and datasets.

Limitations and Future Work

While the paper clearly demonstrates that RLVR harms disagreement modeling, the authors note that their qualitative analysis cannot fully explain some of the more erratic behaviors observed. For instance, why few-shot steering drastically improves performance on some datasets (like GHC) while completely tanking it on others (like HelpSteer2) remains an open question. Additionally, the study focuses entirely on binary classification setups. Future work will need to explore how these dynamics play out in multi-class settings and investigate more token-efficient ways to distill human disagreement into language models.

Summary updated

Abstract

Variation in human annotation (i.e., disagreements) is common in NLP, often reflecting important information like task subjectivity and sample ambiguity. Modeling this variation is important for applications that are sensitive to such information. Although RLVR-style reasoning (Reinforcement Learning with Verifiable Rewards) has improved Large Language Model (LLM) performance on many tasks, it remains unclear whether such reasoning enables LLMs to capture informative variation in human annotation. In this work, we evaluate the influence of different reasoning settings on LLM disagreement modeling. We systematically evaluate each reasoning setting across model sizes, distribution expression methods, and steering methods, resulting in 60 experimental setups across 3 tasks. Surprisingly, our results show that RLVR-style reasoning degrades performance in disagreement modeling, while naive Chain-of-Thought (CoT) reasoning improves the performance of RLHF LLMs (RL from human feedback). These findings underscore the potential risk of replacing human annotators with reasoning LLMs, especially when disagreements are important.
BibTeX
@inproceedings{ni2026can,
  author = {Jingwei Ni and Yu Fan and Vilém Zouhar and Donya Rooein and Alexander Hoyle and Mrinmaya Sachan and Markus Leippold and Dirk Hovy and Elliott Ash},
  title = {{Can Reasoning Help Large Language Models Capture Human Annotator Disagreement?}},
  booktitle = {EACL (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-long.3},
}
Figure