Learning Perceptual Metrics From Unreliable Raters at Scale

Image quality metrics are judged by how well they agree with people, and the people themselves are noisy. Large perceptual benchmarks collect millions of "which of these two images looks better?" clicks from crowd workers. They then collapse those clicks into one score per image using rules that treat every rater as equally careful, and they train or validate metrics against that score. When some raters click at random, the target the metric is fit to is already wrong. Part of the well-known gap between metric scores and human judgment may therefore be a gap between metrics and a corrupted stand-in for human judgment. We propose to train a deep perceptual metric jointly with a model of rater reliability. Expectation-Maximization (EM) is an algorithm for fitting models with hidden variables. It alternates between guessing the hidden variables and refitting the parameters. Here it repeatedly estimates, for every comparison, the probability that the rater was paying attention, and the metric network learns mostly from the comparisons that probably were attentive.

The key observation is about where this matters. If each image is compared hundreds of times, existing EM methods can estimate image scores and rater reliability directly, with no network involved. Real preference data for generated images is sparse: most images are seen once or twice. In that regime, per-image scores cannot be estimated at all, and rater reliability can only be inferred if a network shares information across visually similar images. We call this sharing "amortization." Making the joint fit cheap at millions of comparisons requires the incremental EM machinery developed for large finite sums. That machinery is the second direction's contribution.

Core Claim

The claim has two parts. When comparisons per image are few, a metric trained jointly with an attentive-versus-random rater model will predict held-out human preferences better than the alternatives. The alternatives are a metric trained on raw clicks, a metric trained with a generic noise-robust loss, and a two-stage pipeline that first cleans the labels with a rater model and then trains the metric. The second part is that an incremental EM schedule makes this joint fit cost about the same as ordinary minibatch training of the metric, rather than requiring a full pass over the data for every reliability update.

The Model and the Algorithm

We adopt the rater model of BBQ. BBQ treats each pairwise judgment by rater $r$ as informed with probability $q_r$ and as a fair coin flip otherwise. Using conjugate priors, it derives a closed-form EM for scalar item scores. On unscreened crowds it reaches 61.4% Top-1 ranking accuracy, where Crowd-BT reaches 32.4% and Bayes-BT 23.6%.

Our change is to replace the free per-image score with the output $f_\phi(x)$ of a quality network with weights $\phi$. We use a logistic link in place of BBQ's exponential race so that the network loss stays standard. The probability that rater $r$ prefers image $a$ over image $b$ is then

$$p(y_{ab}=1 \mid r) = q_r\,\sigma\big(f_\phi(a)-f_\phi(b)\big) + \tfrac{1-q_r}{2},$$

where $\sigma$ is the logistic function.

In the E-step, the responsibility $\gamma_i$ of comparison $i$ is the posterior probability that comparison $i$ was informed. It equals the first term divided by the whole expression, evaluated at the observed outcome. The M-step for reliability has a closed form: $q_r$ is updated to the posterior mean under a Beta$(\alpha,\beta)$ prior, which needs only the sum of $\gamma_i$ over rater $r$'s $n_r$ comparisons. The M-step for the network is a gradient step on a Bradley–Terry loss in which each comparison is weighted by $\gamma_i$. Bradley–Terry is the standard model that turns score differences into preference probabilities.

The schedule follows incremental EM as analyzed by Karimi et al. and Fort, Gach and Moulines. We store the most recent $\gamma_i$ for every comparison. For a million comparisons this is a million floats, so the $O(N)$ memory cost these methods are known for does not bind here. Each minibatch refreshes only its own responsibilities and updates the per-rater sums by the difference. We also test a SPIDER-EM-style variant. It periodically recomputes all responsibilities with the current network to remove drift in the stored statistics. That method proves $n+\sqrt{n}\,O(\epsilon^{-1})$ sample complexity for curved exponential families.

We do not claim those guarantees here. The network update is a partial, non-convex M-step outside the exponential-family setting, which is exactly the open limitation the file attributes to SPIDER-EM. Convergence is therefore an empirical question in this proposal.

We deliberately do not model clusters of raters with different tastes. EM-DPO proves that binary comparisons cannot identify mixtures of preference types. MoPLEx shows that Plackett–Luce mixtures of $k$ components are unidentifiable for rankings of length $m\le 2k-1$. Both results constrain our design. Our mixture is different: one of its two components is a known uniform distribution, so the only unknown per rater is a single reliability $q_r$. This is identifiable from pairwise data provided each rater makes enough judgments, and we verify that on synthetic data before relying on it.

Research Program

The work has four stages. First, we implement the joint model with three backbones: a CLIP ViT-L/14 scorer initialized as in Human Preference Score, which fine-tunes CLIP on Discord A/B choices; LPIPS-Alex; and the 5.5M-parameter BiRQA. Second, we construct controlled corruptions of real comparison data. We inject synthetic random-clicking raters at rates from 0% to 50% and thin the data to between 1 and 50 comparisons per image. We know the true reliability of every rater in these settings. Third, we measure where the joint model's advantage over the two-stage pipeline appears and disappears. Fourth, we measure wall-clock cost against full-batch alternation.

GDBC is the closest prior work in spirit. It already alternates EM label calibration with network training for noisy Mean Opinion Scores (MOS), the averaged 1–5 ratings used by most quality datasets. It cuts the error between cheap and expensive MOS by 64–77%, and it names computational cost as its limitation. Our work differs in three ways. It handles pairwise judgments and per-rater reliability rather than scalar MOS bias. It updates reliability incrementally instead of with full passes. And it targets generative distortions, which GDBC never tested.

Baselines

The naive reference is the metric backbone trained with an unweighted Bradley–Terry loss on every comparison, treating all raters as equally careful. This is the implicit assumption behind Elo aggregation in PIPAL. PIPAL collected 1.13 million pairwise judgments over 29,000 distorted images and showed that pixel metrics are anti-correlated with humans on GAN outputs.

The strongest existing competitor is the two-stage pipeline. It runs BBQ's closed-form EM on the comparisons, then trains the same backbone either on BBQ's per-image scores or on comparisons weighted by BBQ's posterior. This is the strongest competitor because it uses exactly our rater model and differs only in whether the network takes part in inference. The reported reference point is BBQ's 61.4% Top-1 accuracy on unscreened IHQ raters (BBQ).

A second strong competitor is the backbone trained with a generic noise-robust loss. We use a Bradley–Terry likelihood mixed with a single global coin-flip rate that is tuned on validation data. This tests whether per-rater modeling matters at all.

Two further variants isolate our ingredients. The first ablation is a direct-gradient version that optimizes the marginal likelihood above with respect to both $\phi$ and $q_r$ by Adam, without EM. It isolates the optimizer from the model. The second ablation is a full-batch version that recomputes every responsibility once per epoch, as in GDBC. It isolates the incremental schedule.

All methods share the following: the same backbone and initialization, the same training comparisons, the same number of network gradient steps, and the same validation-tuning budget of 20 configurations. Tuning covers the learning rate and, where they apply, the Beta prior or the global noise rate.

Beating the two-stage pipeline means statistically significant gains in held-out preference accuracy and in rank correlation with the clean reference, at the sparse settings of one to five comparisons per image. We also require at least parity when coverage is dense. Matching the full-batch variant's accuracy at a fraction of its wall-clock time is the separate scalability criterion.

On PIPAL, the reference correlation for LPIPS-Alex on GAN distortions is an SRCC of 0.4862, rising to 0.5135 with BlurPool (PIPAL). On the HPS Discord data, the reference is HPS at 43.5% choice prediction versus 33.1% for CLIP (Human Preference Score). The file reports no result for BiRQA on PIPAL, so we will establish that baseline ourselves.

Why not just…?

Why not just clean the labels first with BBQ and then train the metric? This is the real threat, and at dense coverage we expect it to tie with the joint model. BBQ estimates a free score per image. When an image appears in only one or two comparisons, that score is driven almost entirely by the prior. Rater reliability is then estimated against nearly uninformative image scores, so a careless rater cannot be distinguished from a careful one who disagrees with the prior. The joint model can tell the two apart, because a rater who contradicts the network's predictions on many visually similar images is evidence of carelessness. The sparsity sweep is designed to settle this. If the two-stage pipeline matches us at one comparison per image on HPS-style data, the joint formulation is not worth its complexity, and we will report that.

Why not just use a robust loss with a single global noise rate? A single noise rate down-weights every comparison equally, so it cannot concentrate trust on reliable raters. The file shows that uniform treatment of heterogeneous raters fails badly: Bayes-BT reaches 23.6% versus BBQ's 61.4% on unscreened raters (BBQ). peerannot finds that voting heuristics underperform per-worker models when there are few workers per task. When noise is spread evenly across raters, a global rate should be close to optimal. Our injection experiments include that condition so that the comparison is fair.

Why not just do gradient descent on the marginal likelihood instead of EM? At the optimum the two coincide, so any gain from EM must come from optimization behavior. BBQ reports that gradient-based Crowd-BT took 15 minutes to converge and was prone to bad local optima, while closed-form EM took seconds. Closed-form reliability updates also remove one learning rate from tuning. The direct-gradient ablation measures this difference. If it matches our method in both accuracy and stability across seeds, we will present the contribution as the model and the amortization, not the EM schedule.

Proposed Experiments

Controlled experiment. We use the HPS Discord data, with 98,807 images and 2,659 identified users, and IHQ, whose split into screened and unscreened raters gives ground-truth reliability. Images are split by prompt into train, validation and test. We inject random raters at 0, 10, 25 and 50%, and thin the data to 1, 2, 5, 10 and 50 comparisons per image. The metrics are held-out choice prediction accuracy on comparisons from clean raters only, and the correlation between the estimated $q_r$ and the true rater type.

PIPAL experiment. On PIPAL we train on thinned, corrupted subsets and report SRCC on GAN distortions against Elo scores computed from the full held-out pool. If PIPAL's per-rater identifiers are not available, the PIPAL experiment is limited to injected raters, and we state this.

Scalability experiment. We report wall-clock time and responsibility drift for the incremental, SPIDER-style and full-batch schedules as the number of comparisons grows from $10^5$ to $10^6$.

Robustness check. We check whether reliability-weighted training reduces metric hacking. We fine-tune a small generator against each learned metric and ask fresh raters whether the generator's gains survive human A/B tests.

Decisive outcomes. The proposal succeeds if the joint model's advantage over the two-stage pipeline grows as coverage gets sparser and vanishes at dense coverage. The incremental schedule must also match full-batch accuracy at a small fraction of its cost. If there is no such crossover, the joint model's extra machinery has not earned its place.