How Much of a Preference Score Is the Protocol? Re-annotating Text-to-Image Benchmarks Under Controlled Instructions

Automatic judges of image quality for text-to-image generators are now trained on large sets of human pairwise choices. These judges, called preference models or reward models, are usually CLIP-style networks fine-tuned to predict which of two images a person picked. Their headline accuracies are routinely compared across papers: 83.3% for HPS v2, 70.5% for PickScore, 65.1% for ImageReward. But each number is measured against human labels gathered under a different protocol. The protocols differ in who rated, what they were told "better" means, how images were displayed, and whether raters were trained. NLP has already shown that simply giving annotators an explicit definition can double the number of errors they report. The vision preference community has never run the equivalent test. When one preference model scores ten points higher than another, we cannot currently tell how much of the gap reflects better perception and how much reflects a better match to one lab's instructions.

We propose a controlled re-annotation study. We take a fixed pool of image pairs drawn from four public preference test sets and relabel every pair under several explicit annotation protocols. The images, prompts, and annotator pool stay fixed, so the only thing that changes is the protocol. This lets us measure three things directly: how far the human "ground truth" moves when only the protocol changes, whether leaderboards of preference models and of generators are stable under that movement, and which quality dimensions each protocol implicitly weights. The deliverable is a protocol-sensitivity estimate for each benchmark, plus a documentation card, adapted from NLP's human-evaluation datasheet, that records the protocol variables that turn out to matter.

Core Claim

Part of what the field calls "perceptual metric misalignment" is not a perceptual failure of the metric. It is disagreement between human labeling protocols that the metric inherits. Our testable prediction has two parts. First, relabeling identical image pairs under different explicit instructions will flip the majority preference on a fraction of pairs noticeably larger than the within-protocol disagreement rate. Second, those flips will be concentrated where quality dimensions conflict, such as a beautiful image with broken anatomy set against a plain but correct one. If the prediction holds, cross-paper preference accuracies are partly protocol artifacts, and a reward model's "failure" on a foreign benchmark can be partly or wholly explained by protocol mismatch. If it fails, and between-protocol agreement matches within-protocol agreement, then the current leaderboards are more trustworthy than their undocumented collection processes suggest. That is also a result worth publishing.

What Prior Work Establishes and Leaves Open

Evidence that protocols move human judgments comes from NLP and video. ReproHum #1018-09 re-ran a data-to-text redundancy evaluation. When annotators were given an explicit definition, they flagged roughly twice as many errors as the original authors. Yet system rankings stayed correlated (Pearson 0.76). This cuts against our hypothesis as much as for it: labels may shift while rankings survive. Our design therefore measures label-level and ranking-level stability separately rather than assuming instability.

For text-to-video, the T2VHE protocol showed that instruction- and example-based training raised video-quality inter-annotator agreement from 0.185 to 0.411. Training changes the labels, not just their noise, but that study never asked whether a trained and an untrained pool rank models differently.

Surveys motivate the concern that such effects go unmeasured. Best practices for the human evaluation of automatically generated text found that agreement was reported in only 12.5% of NLG papers. The Great Misalignment Problem found that only 2 of 10 audited papers evaluated what their method actually modeled. The Human Evaluation Datasheet supplies a documentation template; we reuse its structure but decide empirically which fields matter for vision.

On the vision side, four datasets supply the images and the models under test. HPS v2 collected choices from 57 professional annotators across nine generator families. Pick-a-Pic used untrained web users. ImageReward used expert raters with an alignment, body-problem, and aesthetics rubric. MPS collected ratings on four separate dimensions. Each paper collected labels once, under one protocol, so none can separate protocol effects from image or prompt effects.

Cross-dataset numbers already hint at the problem, but they confound protocol with image source and prompts. For example, HPS v2 scores 65.7% on the ImageReward test split versus 83.3% on its own. UniSim shows HPS v2 falling below vanilla CLIP on Text-2AFC (about 76% versus 76.8%). Stable Preference shows 1.5–1.9% accuracy drops under imperceptible crops. Both constrain our design: display handling must be controlled, or we will measure crop sensitivity instead of instruction sensitivity.

HYPE (HYPE) is the closest methodological ancestor. It standardized human evaluation of realism with psychophysical controls. It addressed unconditional generators and a single realism question, not how competing definitions of "better" reshape preference labels for text-to-image models.

Research Program

We sample a pool of 2,000 image pairs, 500 from each of the test splits of HPD v2, Pick-a-Pic, ImageReward, and MHP. We keep each pair's original label. We stratify sampling by how strongly the original raters agreed and by whether MPS predicts conflicting dimension orderings within the pair, which lets us test where flips concentrate.

Every pair is relabeled under five protocols by the same recruited pool. Each pair gets at least five independent judgments per protocol, and no annotator sees the same pair twice across protocols.

The protocols are as follows. The unguided protocol asks only which image is better for the prompt. The holistic-expert protocol reconstructs HPD v2's instructions as closely as that paper's description allows; the fact that reconstruction requires guesswork is itself a documentation finding. The rubric-first protocol follows ImageReward's ordering of alignment, then structural defects, then aesthetics. The decomposed protocol has raters score MPS's four dimensions before giving an overall choice. The trained protocol adds T2VHE-style worked example critiques before any rating.

A crossed display factor presents each pair either at native resolution and aspect ratio or at a fixed size with a standardized center-crop. This separates instruction effects from the framing sensitivity documented by Stable Preference.

We use three analyses.

The first is a variance decomposition of pair-level preference probability into pair, protocol, display, and annotator components. Its key quantity is the between-protocol flip rate compared against a within-protocol split-half flip rate. Because the within-protocol rate is estimated by the same resampling, it forms the natural null.

The second is ranking stability. Under each protocol's labels we recompute the accuracy of PickScore, HPS v2, ImageReward, MPS, Stable Preference, and vanilla CLIP, and report the Kendall rank correlation of the model leaderboard across protocols. We do the same for the generator families contained in HPD v2.

The third recovers implicit weightings. For each protocol, we fit a logistic model predicting its majority label from MPS's per-dimension score differences and our own raters' dimension ratings. The fitted weights make visible, for instance, whether unguided raters weight aesthetics far above alignment.

The byproduct is a short protocol card for visual preference data. It keeps only the HEDS-style fields whose variation produced measurable label shifts in our study, so the reporting overhead that has slowed HEDS adoption is justified by evidence.

Baselines

The naive reference point is the status quo: take each preference model's published accuracy on its home benchmark at face value and treat cross-paper differences as capability differences. We tabulate these numbers exactly as reported. The strongest reported figure is HPS v2 at 83.3% on HPD v2 (HPS v2), against PickScore at 79.8% and ImageReward at 74.0% on the same split. PickScore reports 70.5% on Pick-a-Pic, and MPS reports 74.24% overall on MHP. Our study "beats" this baseline if the leaderboard implied by these numbers changes order under at least one explicit protocol on our fixed pool, or if protocol explains more label variance than annotator identity.

The strongest existing competitor for explaining cross-benchmark disagreement is ordinary cross-dataset evaluation of off-the-shelf models, of which the reported 65.7% for HPS v2 on the ImageReward split is the canonical example. We run it fairly by scoring all six models with their released checkpoints on the same 2,000 pairs, against the original labels and against every protocol's labels, with no fine-tuning of any model. This gives every model an identical data and compute budget.

The ablation that isolates our key ingredient holds the protocol fixed and swaps the annotator pool. Specifically, it compares our pool's unguided labels against Pick-a-Pic's original web-user labels, and our holistic-expert labels against HPD v2's originals. If pool swaps move labels as much as protocol swaps do, the effect is about who rates rather than what they are told, and the protocol card must emphasize recruitment over instructions. A second ablation drops the display factor to confirm that instruction effects survive when framing is held constant.

Why not just…?

Why not just report inter-annotator agreement within each existing dataset? Within-protocol agreement already exists, for example 78.1% individual-rater accuracy on HPD v2 and 68.0% for experts on Pick-a-Pic. However, it measures noise around one protocol's definition of quality, not the distance between definitions. Two protocols can each be internally consistent and still disagree systematically. ReproHum's doubled error count arose between consistent annotator groups. Our between-protocol versus within-protocol flip comparison is precisely the quantity that agreement statistics cannot supply.

Why not just use a multi-dimensional score like MPS and stop worrying about protocols? This is the most serious alternative. If protocols differ only in how they weight a fixed set of dimensions, a decomposed scorer with a per-protocol weighting would reconstruct every protocol's labels, and our study would reduce to estimating those weights. The implicit-weighting analysis tests this directly. If MPS dimension differences, reweighted per protocol, predict each protocol's labels about as well as the split-half human ceiling, we will report that decomposition suffices. The contribution then becomes the measured weights and the demonstration that single-scalar leaderboards encode them silently. If a residual remains after reweighting, particularly under the trained protocol, protocols change what raters perceive and not just how they combine it.

Why not just train one robust preference model on all datasets pooled? Pooling, as UniSim does across similarity tasks, averages over protocols without identifying them. UniSim itself reports that it remains bounded by label noise in its source datasets. A pooled model would score well against a mixture of protocols while hiding which protocol any given downstream user needs. Our relabeled pool can evaluate a pooled model per protocol, so this alternative becomes one more row in the ranking-stability analysis rather than a replacement for it.

Proposed Experiments and Decisive Checks

The primary experiment is the full relabeling: 2,000 pairs, five protocols, two display conditions, and five judgments per cell, roughly 100,000 judgments. Raters are screened with control pairs of obvious quality difference, in the manner of HYPE's qualification task, and paid a fixed hourly wage.

The first decisive check is whether the between-protocol majority flip rate exceeds the within-protocol split-half flip rate, with bootstrap confidence intervals over pairs and annotators. No reported result was found on Lacuna for protocol-induced flip rates on any text-to-image preference dataset, so no prior number exists to beat. The null is our own split-half rate.

The second check is whether the Kendall correlation of the six-model leaderboard across protocols falls meaningfully below one. ReproHum's preserved rankings are the explicit counter-hypothesis, and a result showing stable rankings despite shifting labels would replicate that finding in vision.

The third check is whether flips concentrate in the stratum where MPS predicts conflicting dimension orderings. This would locate protocol sensitivity at the aesthetic-versus-structure trade-off that scalar scores collapse.

A secondary experiment tests the practical consequence. We fine-tune only the final layers of a CLIP ViT-L/14 scorer on labels from one protocol and evaluate it on held-out pairs from every other protocol. We compare the resulting transfer matrix with the cross-dataset matrix of the original benchmarks. If protocol mismatch alone reproduces a large share of the known cross-dataset accuracy drops with images held constant, then those drops have been misread as perceptual failures of the metrics.