Incentivizing High-Quality Human Annotations with Golden Questions

Abstract

Ensuring high-quality human annotations is a critical challenge in training Large Language Models (LLMs). This paper addresses the "moral hazard" where paid annotators minimize effort to maximize earnings. By modeling the interaction as a Stackelberg game, the authors demonstrate that strategic behavior significantly slows the convergence of quality monitoring. They propose a "golden question" selection strategy using Reward Models to identify high-certainty, format-consistent samples. Empirical results from social experiments on Prolific show that these golden questions outperform standard attention checks in identifying high-quality contributors.

Introduction

Large Language Models (LLMs) rely heavily on human-annotated data for supervised fine-tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF). However, ensuring the quality of this data is notoriously difficult. Unlike objective tasks like image classification, language preferences are often subjective and heterogeneous.

Paid annotators often face a conflict of interest: they want to maximize their earnings while minimizing effort. This leads to "cheating" behaviors, such as random clicking or using AI to generate responses. Current systems often rely on an "honor code" or simple "instructed manipulation checks" (e.g., "Please select option A to prove you are reading"). The authors argue these are insufficient because strategic annotators can easily spot these checks and only exert effort on the "test" questions while slacking on the actual data.

Theoretical Framework and Incentive Design

The researchers model the relationship between a company (the principal) and an annotator (the agent) as a Stackelberg game. In this model, the principal first commits to a payment structure, and the agent then chooses an effort level to maximize their utility.

The core of the incentive system is a binary contract. The principal estimates the annotator's unobserved effort level $\theta_a$ using Maximum Likelihood Estimation (MLE) based on their performance on a small set of $n$ "golden questions."

Expand

Source, p. 5

As shown in Algorithm 1, the annotator receives a base salary $W_0$. If their performance on the golden questions passes a specific hypothesis test $\Psi$, they receive an additional bonus $W_b$. This structure forces the annotator to weigh the cost of effort against the probability of receiving the bonus.

The Cost of Strategic Behavior

Using the "log-trick" and Berry-Esseen-type bounds, the authors analyze the variance of the outcome. They discover a "strategic rate" of convergence: $\Theta(1/\sqrt{n \log n})$. This is much slower than the exponential rate found in traditional, non-strategic hypothesis testing. This finding proves that the annotator's ability to adjust their effort to "just barely pass" the test makes monitoring significantly harder than previously assumed.

Golden Question Selection

To implement this framework effectively, the authors propose a Reward Model (RM)-based algorithm to select golden questions. Effective golden questions must satisfy two criteria:

1. High-Certainty: They must have a near-certain "correct" answer. If the golden questions are ambiguous, the noise makes it impossible to distinguish a lazy annotator from a thoughtful one with a different opinion.
2. Camouflage: They must be similar in format to normal tasks. If an annotator can distinguish "test" questions from "real" questions, they will only exert effort on the former.

Experimental Results

The authors validated their selection criteria using six prominent human preference datasets, including OpenAI’s Summarize from Feedback and Anthropic’s HH-RLHF. Using a Skywork-Reward model to pick the top 10% most certain samples resulted in over 90% agreement with human labels across most datasets.

In social experiments with 90 human annotators on the Prolific platform, "Real" golden questions (selected by the algorithm) were significantly better at distinguishing annotator quality than standard "Instructed" attention checks. The accuracy gap between "Correct" and "Incorrect" annotator groups was 11.1% for real golden questions, compared to only 7.0% for instructed checks. Annotators who passed the real golden questions demonstrated significantly higher quality on the rest of the unmonitored dataset.

Conclusion and Limitations

This work highlights the fundamental limits of monitoring strategic agents and provides a robust framework for improving data quality in LLM training. However, limitations remain. While the paper focuses on high-certainty questions, much of LLM alignment involves highly subjective "gray areas." Additionally, the model assumes consistent behavior across a batch, whereas real-world annotator effort may fluctuate due to fatigue or learning. Future work will explore extending these incentives to more complex tasks like long-form reasoning and multi-modal annotations.

@misc{liu2025incentivizing,
  author = {Shang Liu and Zhongze Cai and Hanzhao Wang and Zhongyao Ma and Xiaocheng Li},
  title = {{Incentivizing High-Quality Human Annotations with Golden Questions}},
  year = {2025},
  eprint = {2505.19134},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.19134},
}