Paying crowd annotators a bonus only when they pass hidden quality checks is becoming the standard defense against low-effort work and copy-pasted chatbot answers. Any such check is a noisy test run on a handful of items. Two equally careful workers can therefore land on opposite sides of the pass line, and one gets paid while the other does not. This proposal asks what that luck does to the next batch of labels. The unlucky worker has two reasons to change behavior. The first is purely self-interested: they may conclude that effort does not pay. The second is social: they may resent earning less than peers who did the same work. The two explanations call for different fixes. The first calls for a less noisy test. The second calls for a contract whose pay differences workers see as earned rather than drawn from a lottery.
We propose a large preregistered annotation experiment that separates these two explanations. The design compares workers who scored just above and just below the bonus threshold, which makes their bonus close to randomly assigned. It also randomizes whether workers can see how peers fared. From the results we build and test a bonus contract that keeps the incentive while shrinking luck-driven differences in pay. Theory does not settle even the sign of the effect, which is a large part of why the experiment is worth running.
Core Claim
Audit-based bonus contracts produce pay dispersion that workers do not control, and today's contract theory treats annotators as isolated self-interested agents who never compare their pay to anyone's. The Stackelberg binary contract in Incentivizing High-Quality Human Annotations with Golden Questions models a single annotator with a scalar effort level. It pays a base wage plus a bonus when the annotator passes a hypothesis test on hidden "golden questions," which are items with near-certain answers. It proves that strategic annotators slow monitoring to a $\Theta(1/\sqrt{n\log n})$ rate. The sentinel-auditing mechanism of Overcoming the Incentive Collapse Paradox injects deliberate AI errors and pays bonuses for catching them. It explicitly assumes a representative worker with homogeneous costs. Neither paper asks how workers react to what their peers receive.
We claim that the effect of an unlucky audit outcome on subsequent effort can be measured. We further claim it can be decomposed into a belief-learning part, present whether or not peers are visible, and a peer-comparison part, present only when they are. We expect the second part to be large enough that equalizing audit exposure improves label quality per dollar over the golden-question contract.
The sign of the peer-comparison part is genuinely open. Inequity aversion is the idea, formalized by Fehr and Schmidt, that people dislike being behind others ("envy," weight $\alpha$) and sometimes dislike being ahead ("guilt," weight $\beta$). Write a worker's utility for a batch as
$$U_i = w_0 + b\,\mathbb{1}[\text{pass}_i] - c(e_i) - \alpha\,\max(\bar{b}_{-i} - b\,\mathbb{1}[\text{pass}_i],\,0),$$
where $w_0$ is base pay, $b$ the bonus, $e_i$ effort with cost $c(e_i)$, and $\bar{b}_{-i}$ the average bonus the worker believes peers received. Read this way, envy makes failing more painful and should raise effort, as in tournaments.
A second tradition predicts the opposite. The mechanisms linking perceived stress to pilots' safety attitudes: a chain mediation effect of job burnout and cognitive flexibility reads burnout through social exchange theory: workers who feel unsupported withhold "extra effort" to punish the system. When Incentives Backfire, Data Stops Being Human argues conceptually that extrinsic pay crowds out careful thinking. Under this reading, a visibly unlucky worker becomes cynical and shirks. The experiment is built so that either sign is a publishable finding, and the contract design follows whichever sign appears.
Research Program
Task and contract. The annotation task reuses the setting of the golden-question paper. Workers make pairwise preference judgments on Anthropic HH-RLHF and OpenAI Summarize-from-Feedback items. Golden questions are selected with a Skywork reward model, whose top-10% most certain items agree with human labels over 90% of the time on most datasets according to that paper. They are camouflaged in the same format as production items. Workers complete several batches, each ending in a bonus decision, so every worker has a "next batch" in which their response to the previous outcome is observed.
Identification. The central estimate uses a regression discontinuity: we compare workers whose golden-question accuracy fell just above versus just below the pass threshold. These workers put in nearly the same effort but received different bonuses. The discontinuity in next-batch quality is crossed with a randomized visibility arm. In the visible arm, workers see, after each batch, what fraction of their cohort earned the bonus and where they ranked. In the hidden arm, they see only their own outcome.
The two components are separated as follows. The discontinuity in the hidden arm estimates belief learning plus any comparison with one's own past. The difference between the visible and hidden discontinuities isolates peer comparison.
A third arm keeps peers visible but adds a plain-language explanation that the test is a random sample and that near-threshold outcomes are partly luck. It tests the observation in The Fairness Fair: Bringing Human Perception into Collective Decision-Making that people's sense of fairness tracks procedure and transparency, not just outcomes.
Measuring the worker. We use multi-dimensional psychological measurement for one specific job: telling apart two reasons a worker's quality might drop after missing a bonus. One is fatigue. The other is resentment.
At intake we estimate each worker's envy parameter behaviorally, from the minimum offer they accept in a one-shot ultimatum game, rather than from a long questionnaire. After each batch we ask two short items adapted from each of two Maslach Burnout Inventory subscales: emotional exhaustion and cynicism. Four items are far short of the full inventory. We accept the loss of psychometric validity to avoid the survey fatigue that makes full batteries impractical in microtask settings.
The prediction is that a peer-driven drop is carried by cynicism and concentrated among workers with high $\alpha$, while fatigue-driven decline tracks exhaustion and time on task. This is a mediation analysis in the style of Zhao et al.'s chain model. Because the bonus is randomized at the threshold, the first link is causal rather than cross-sectional, which answers the cross-sectional confounding problem that paper acknowledges.
The contract we build. If peer comparison lowers effort, we replace the standard contract with an equalized-exposure contract that has the same expected cost. Every worker receives the same number of golden questions per batch, placed in the same positions relative to batch length. The bonus is paid on a shrinkage estimate of accuracy pooled across the worker's batches, not on a single-batch test. This reduces the share of pay dispersion driven by sampling luck while keeping pay tied to sustained effort.
The dynamic comparison follows the modelling idea in Inequity aversion improves cooperation in intertemporal social dilemmas. There, agents compare temporally smoothed reward traces of themselves and peers, and a delayed comparison signal was shown to break cooperation. We fit a smoothed-trace version of the utility above to the batch-by-batch data, to see whether workers react to cumulative disparity or only to the latest outcome.
If instead envy raises effort, the design goal flips. The contract should make relative standing visible, and the equalization fix becomes unnecessary. We would report that result as such.
A secondary regime. The same design runs in a human-verifies-AI condition. Here workers check AI pre-labels and earn sentinel bonuses, as in the incentive-collapse paper. In that setting luck enters through how many sentinels a worker happens to be shown, so the equalized-exposure fix translates directly into equal sentinel counts per worker.
Baselines
The naive reference point is flat pay with no bonus, at the same expected total cost per worker as the bonus arms. It removes luck-driven dispersion entirely but restores the effort-minimization problem that motivated bonus contracts. It shows how much the incentive is worth in the first place.
The strongest existing method is the golden-question binary contract of Incentivizing High-Quality Human Annotations with Golden Questions. We run it with its own Skywork-based golden-question selection, its test, and its bonus size. It is the only contract in the file that was tested on real crowd workers and designed against strategic shirking. The paper reports that its camouflaged golden questions separated diligent from non-diligent annotators by an 11.1% accuracy gap on Prolific with 90 annotators, versus 7.0% for instructed attention checks. It is also the contract whose single-batch threshold creates the luck we study. For fairness, every arm uses the same item pool, the same golden-question selection, the same number of golden items per worker in expectation, and the same expected payout. Beating it means higher accuracy on held-out production items per dollar spent, with no rise in attrition.
A second strong competitor is the same contract with the golden-question count doubled. It reduces test noise by spending more budget, and it sets the price of simply buying away luck. The equalized-exposure contract must match its quality at lower cost to count as a win.
We also run post-hoc reliability reweighting. We collect data under the standard contract and then aggregate it with EffiARA's recursive reliability weights, as described in Efficient Annotator Reliability Assessment with EffiARA. On RUC-MCD that method raised downstream F1-macro from 0.691 under majority voting to 0.740. This baseline tests whether degraded labels can simply be downweighted after the fact.
Two ablations isolate the key ingredient: equal exposure without cross-batch shrinkage, and shrinkage without equal exposure. There is no reported result found on Lacuna for how peer bonus visibility affects annotation quality, so the peer-comparison effect has no prior number to beat. It is a quantity to estimate.
Why not just…?
Why not just pay a generous flat wage and remove the bonus? Work cited in the file argues that higher flat pay does not reliably raise quality and can attract opportunistic throughput-maximizers (Benoît Sagot et al.; Joanna Mrozinski et al.). Still, this alternative is a real threat when dispersion effects turn out to be large, so it is a full arm of the experiment rather than a strawman. If flat pay matches the equalized contract on held-out accuracy per dollar, we will conclude that the bonus is not worth its fairness cost in this task.
Why not just hide everyone's bonus outcomes? Hiding removes explicit peer comparison, and the hidden-visibility arm measures exactly how much that buys. Hiding cannot remove the self-interested belief-learning channel: a worker who misses the bonus after trying hard still learns that effort looks unrewarded. On platforms, workers also compare notes outside the interface. If the hidden-arm discontinuity is as large as the visible one, hiding is sufficient against peer effects and our equalization argument rests only on belief learning. The design reports that case directly.
Why not just explain that the test is noisy? Hosseini's evidence that procedural transparency shapes perceived fairness makes this the cheapest plausible fix. It gets its own arm. If explanation alone erases the visible-arm discontinuity, it beats our contract on simplicity, and we will recommend it.
Why not just downweight bad labels afterward with reliability models? Reweighting, as in EffiARA or the group-bias model of Toward Annotator Group Bias in Crowdsourcing, treats a worker's decline as a fixed trait. That group-bias model reached 0.968 F1 on Wikipedia Detox Personal Attack against 0.955 for majority voting. Reweighting discards paid-for labels rather than preventing the decline. The quality-per-dollar comparison against the reweighting baseline shows whether prevention or cleanup is cheaper.
Isn't any drop after a miss just rational learning, not inequity aversion? Yes, possibly, which is why the design never attributes the hidden-arm discontinuity to social preferences. Only the difference between the visible and hidden arms, together with its concentration among high-$\alpha$ workers, counts as evidence for peer comparison.
Proposed Experiments and Decisive Checks
We plan roughly 1,500 workers so that enough fall within a narrow band around the threshold for the discontinuity estimates. This is an order of magnitude beyond the 90 in the golden-question study. Workers are assigned across flat pay, standard contract with peers visible, standard contract with peers hidden, visible plus procedural explanation, doubled golden questions, and the equalized-exposure contract. They are recruited on the same platform used by the golden-question study.
The primary outcome is next-batch accuracy on camouflaged golden questions. Secondary outcomes are agreement on held-out non-golden items with labels from a separately recruited, high-diligence reference pool; time on task; attrition; and the four burnout items.
The first decisive check is whether the discontinuity in next-batch accuracy differs between visible and hidden arms, and in which direction. The second is whether that difference grows with the elicited $\alpha$ and runs through cynicism rather than exhaustion. The third is whether the equalized-exposure contract beats both the standard contract and the doubled-golden-question contract on held-out accuracy per dollar.
A null result on the first check is informative. It would say that existing contracts are safe to ignore peer effects, and the programme would then reduce to choosing the cheapest noise reduction. The sentinel-auditing replication on AI-pre-labelled items then tests whether the answer carries over to human verification of AI output.
References
- Incentivizing High-Quality Human Annotations with Golden Questions
- Overcoming the Incentive Collapse Paradox
- The mechanisms linking perceived stress to pilots’ safety attitudes: a chain mediation effect of job burnout and cognitive flexibility
- Position: When Incentives Backfire, Data Stops Being Human
- The Fairness Fair: Bringing Human Perception into Collective Decision-Making
- Inequity aversion improves cooperation in intertemporal social dilemmas
- Efficient Annotator Reliability Assessment with EffiARA
- Un turc mécanique pour les ressources linguistiques : critique de la myriadisation du travail parcellisé (Mechanical Turk for linguistic resources: review of the crowdsourcing of parceled work)
- Collecting a Why-Question Corpus for Development and Evaluation of an Automatic QA-System
- Toward Annotator Group Bias in Crowdsourcing