Asking at the Right Moment: Active Image-Quality Labeling with Drifting Human Raters

Models that judge image quality without a clean reference image learn from human opinion scores, and collecting those scores is slow and expensive. Active sampling cuts the cost by showing raters only the most informative items, usually near-ties between two images of similar quality. These are also the judgments people get wrong most often, and people get them wrong more often at particular moments: before they have warmed up, after they have grown tired, or right after seeing an unusually good or bad image. Current active methods for image quality treat every rating as equally trustworthy whenever it was given. We propose to model each rater's reliability as a quantity that changes over the session, estimate it without gold-standard answers by occasionally re-showing images the rater has already scored, and use that estimate to decide what to ask. The key decision is to place the hardest comparisons at the points in a session where the rater is currently reliable.

The claim is that "what to ask" and "when to ask it" cannot be separated in subjective labeling. Hard items make errors more likely, and an unreliable rater state makes them more likely still, so the two effects should compound. If they do, then a sampler that ignores rater state wastes much of its effort on its most valuable queries, and correcting labels after the fact recovers only part of that loss. The proposal tests whether this interaction exists. If it does, we build a sampler that exploits it and measure the gain in label efficiency and in the resulting quality model.

What exists and what is missing

The closest work sits on two sides of the problem. Active Sampling for Subjective Image Quality Assessment proposes HY-ACT. It treats each image's quality as a Gaussian variable, links absolute scores and paired comparisons through Thurstone models, and picks pairs by expected information gain, which favors near-ties with high uncertainty. It assumes observers are static. Active Fine-Tuning From gMAD Examples Improves Blind Image Quality Assessment finds image pairs on which a blind quality model and reference-based metrics disagree most, collects human ratings on those pairs, and fine-tunes on them as ground truth. Its authors report that the model overfits after several rounds. We read that as consistent with, though not proof of, noisy labels on hard pairs being absorbed as truth. Both papers motivate this proposal. Neither models the rater.

On the rater side, Bidirectional Active Learning with Gold-Instance-Based Human Training (BALT) tracks how annotator competence improves by inserting gold instances with explanations, and it relabels old items once competence rises. Subjective quality has no gold answers, and BALT leaves fatigue to future work. Temporal-aware Language Representation Learning From Crowdsourced Labels (TACMA) down-weights labels that conflict with a worker's last 7 to 10 labels on similar items. It works passively, on text and speech, with a fixed window. SoQal decides when to query a cardiologist and when to pseudo-label, and Active Learning with a Noisy Annotator filters noisy labels and resamples around them. Both assume noise is constant over time and are built for classification.

We reuse three techniques from this work: information-gain pair selection from HY-ACT, the hard-pair mining loop from gMAD, and relabeling as a legitimate action from Re-Active Learning. What we add is a model of rater state estimated without gold labels and an acquisition rule that conditions on it.

The evidence that the gap matters comes from active learning more broadly. Items near the decision boundary raise human error rates (Active Learning for New Domains in Natural Language Understanding). Systematic coder bias can push active learning below random sampling (Evaluating the Impact of Coder Errors on Active Learning). Judges also follow a trajectory within a session, starting unfamiliar, peaking once warmed up, and then declining (Fomicheva et al.).

Core Claim

Consider rater $r$ giving response $y$ at session position $t$ on a query with difficulty $d$, where difficulty is the model's predicted score gap for a pair or the posterior spread for a single image. We model the response as the latent quality plus a rater-state term:

$$ y = q + b_r(t) + \gamma_r\, (q_{\text{prev}} - \bar q) + \epsilon, \qquad \epsilon \sim p_r(\cdot \mid t, d). $$

Here $q$ is the latent quality of the item (or the quality difference for a pair), $b_r(t)$ is a slowly drifting bias, $\gamma_r$ captures anchoring on the previous stimulus of quality $q_{\text{prev}}$ relative to the session mean $\bar q$, and $p_r$ is the residual noise distribution. The noise is allowed to depend on both session position and difficulty.

The testable prediction is an interaction: the noise spread grows with $d$ faster in some rater states than in others. When that holds, expected information gain should be computed under the current rater state, and the sampler should give the hardest items to raters whose current state is good. Easy items, relabels, and consistency probes fill the remaining slots.

The rater state is estimated from probe trials, which re-show images the same rater has already rated, placed at random positions. Disagreement between a rater's first and second answer on the same image measures that rater's own variability over time, with no ground truth needed. This separates two sources of spread. Disagreement across different raters on a stable image reflects real differences in taste. A rater contradicting themselves reflects drift. Aggregate opinion scores and their standard deviations mix the two.

The residual distribution $p_r$ is where non-parametric density modeling enters. We estimate it with kernel density estimates pooled across raters and conditioned on binned $(t, d)$, instead of assuming a Gaussian. The motivation mirrors Mory et al., who show that distributions with identical mean and variance can differ in structure. A lapse-plus-noise mixture, for example a tired rater who sometimes clicks at random, has a heavy-tailed shape that a Gaussian with matched variance will misweight. We treat this as a hypothesis with its own ablation, not as a premise. If a Gaussian whose variance depends on $t$ and $d$ performs as well, we drop the kernel estimate.

Research Program

The first stage is measurement: does exploitable drift exist in image-quality rating at all? We will first check whether the released raw ratings for KonIQ-10k include worker identity and presentation order. If they do, we fit the state model to them directly. If they do not, which is likely, we run our own study. Raters in controlled lab conditions and on a crowdsourcing platform score images drawn from KonIQ-10k and gMAD pairs mined from the Waterloo Exploration Database, following the gMAD protocol. Probe repeats make up about ten percent of trials, and presentation order is logged.

From these data we estimate four things: how probe consistency changes with session position, the anchoring coefficient $\gamma$, how error rate depends on difficulty, and the difficulty-by-state interaction. This stage decides whether the rest of the program is worth running.

The second stage builds the state-aware sampler. HY-ACT's pair-selection step is kept, but its observation model is replaced by the rater-specific, state-dependent likelihood above. At each step the sampler chooses among four actions: a new hard pair, a new easy item, a relabel of an existing item whose label was given in a poor state, or a probe. It picks the action with the highest expected information gain per unit of rater time. The rater state is updated online with a simple filter over $b_r(t)$ and the noise scale.

The third stage closes the loop with a quality model. We rerun the gMAD active fine-tuning procedure over several rounds with labels produced by each collection strategy. This tests whether state-aware labels reduce the overfitting across rounds that the original authors report.

Baselines

The naive reference is random sampling of single-image absolute scores, averaged into opinion scores with no rater model. It is what most benchmark collections do, and it sets the floor for label efficiency.

The strongest existing sampler is HY-ACT from Active Sampling for Subjective Image Quality Assessment. It reports reaching a Kendall's τ of 0.90 with 28 observations per image on LIVE, against 67 for random-graph methods. We run its published procedure on the same image pools and the same rater population with the same total rater-minutes. Its scale hyperparameters get the same tuning budget as ours, set on a held-out tuning session. Beating it means reaching τ = 0.90 against the dense reference with meaningfully fewer observations per image, or reaching higher τ at a matched budget, with confidence intervals from bootstrapping over raters.

The strongest competitor on the rater side is HY-ACT plus a static per-rater reliability weight, a Dawid–Skene-style correction. It captures consistent differences between people without modeling time. TACMA reports that Dawid–Skene fails when reliable workers become inconsistent (Temporal-aware Language Representation Learning From Crowdsourced Labels), but that evidence comes from text, so here the comparison has to be run directly.

We also transplant TACMA's fixed-window reweighting. Its "similar items" are defined as nearby images in the feature space of the quality model, and it is applied to HY-ACT's labels after collection.

For the downstream model, the reference is the gMAD active fine-tuning result of an SRCC of 0.818 and a PLCC of 0.823 on the gMAD test set L(1) (Active Fine-Tuning From gMAD Examples Improves Blind Image Quality Assessment). We reproduce its pipeline under our own labeling budget, because absolute numbers from a different rater pool will not transfer. We also report cross-dataset SRCC on KonIQ-10k, LIVE, CSIQ, and TID2013. MUSIQ's reported SRCC of 0.916 on KonIQ-10k (MUSIQ) is context for what a strong model reaches with full labels.

Three ablations isolate the key ingredients. The first keeps the state model for post-hoc label correction but schedules queries as HY-ACT does, which separates the scheduling gain from the correction gain. The second schedules by state but replaces the kernel residual with a heteroscedastic Gaussian. The third removes the probes and infers state only from cross-rater disagreement.

Why not just…?

Why not just cap session length and add breaks? This is the most serious alternative, and it is standard practice in subjective testing. If drift is mainly fatigue that grows steadily over the session, breaks remove most of it at almost no modeling cost. Breaks do nothing about anchoring on the previous image or about warm-up at the start of a session, and they do not concentrate hard items where the rater is reliable. We include HY-ACT with sessions capped at the length where probe consistency begins to drop in stage one, matched on rater-minutes and counting break time against the budget. If this baseline matches our sampler, we will report that scheduling adds nothing beyond breaks, and the contribution reduces to the measurement study.

Why not just use a static per-rater reliability weight? It is cheap and captures the large differences between careful and careless raters. It cannot represent a rater who is reliable at minute five and unreliable at minute forty. Stage one tests this directly: if within-rater probe consistency does not vary with session position beyond noise, static weighting is sufficient and we stop there.

Why not just collect more ratings per image? More raters average out noise, but only noise that is independent across raters. Anchoring effects caused by presentation order, and the higher error rate every rater shows on hard pairs, are correlated, so averaging reduces them more slowly than independent noise. The label-efficiency curves settle this, since random sampling with many ratings per image is exactly the naive baseline. The honest cost comparison is observations per image at a fixed τ.

Why not just skip human labels and use a vision-language scorer? Q-Align and LIQE still depend on human-scored data for alignment, and LIQE's authors note failures on unseen artifacts produced by generative models. New artifact types will keep requiring fresh human labels, and those labels are the ones this proposal aims to make cheaper and more reliable.

Proposed Experiments

The decisive first check is the stage-one interaction test. We fit a mixed model of probe disagreement with fixed effects for session position, difficulty, and their product, plus random slopes per rater. The program continues only if the interaction term is reliably non-zero and position explains a meaningful share of each rater's variability. Otherwise we publish the null result and stop.

The dense evaluation reference is a separate pool of raters who each rate in short, randomized sessions, giving many ratings per image. It is expensive, but it is only needed for evaluation.

The main comparison plots τ against the dense reference as a function of rater-minutes for every baseline and ablation, on the KonIQ-10k subset and on the gMAD pairs. We predict the largest gap on the gMAD pairs, where difficulty is highest. The downstream comparison runs three rounds of gMAD fine-tuning per labeling strategy and tracks SRCC on the gMAD test set and on held-out benchmarks, to see whether the drop across rounds shrinks.

A controlled simulation with ground truth complements the human studies. Latent qualities are taken from LIVE and TID2013 opinion scores, and simulated raters follow the fitted drift parameters. This checks that the sampler's gains track the size of the drift, and that the gains vanish when drift is set to zero, which guards against the method helping for unrelated reasons.