Image quality models predict how good a photo looks to people. They are trained on small sets of human ratings, usually a few thousand images, each labeled with a Mean Opinion Score (MOS), the average of many raters' scores. Because ratings are scarce, teams increasingly use human-in-the-loop refinement: a model drafts scores for unlabeled images, a person corrects the draft, and the corrected labels are fed back to retrain the model for the next round. Now suppose one rater invokes the right to be forgotten, the legal right under laws such as GDPR to have one's personal data deleted. That rater's judgments are personal, since they encode one person's perception. Their influence is also spread out. It sits in their own ratings, in the pseudo-labels propagated from their corrections, and in the drafts that later raters saw. Later raters' edits are anchored on those drafts, so the departed rater shapes labels they never touched. Retraining from scratch on the "remaining" labels does not remove that last channel, because the remaining labels were themselves shaped by the deleted person.
This note proposes a deletion method built for this setting. It rests on one observation. In subjective regression, the counterfactual label is often computable directly: the score an image would have had without rater $k$ is just the mean over the other raters. Classification unlearning instead pushes outputs toward a uniform guess, and the literature warns that this can collapse useful representations. We replace that with fine-tuning toward known leave-one-rater-out targets. We add a provenance record that traces every propagated label and every shown draft back to the corrections that produced it. Finally, we measure and correct for anchoring, the second-order channel. Fast networking enters only as a narrow, falsifiable systems question about whether deletion can happen while the annotation tool is live.
Core Claim
Deleting a rater from a human-refined quality model takes three steps. First, recompute the targets directly wherever the rater contributed. Second, use a provenance graph to find the propagated labels and drafts their corrections touched. Third, adjust later raters' edits by an anchoring coefficient measured from people. Together these steps should yield a model whose predictions match a "never-included" reference far more closely than either retraining on the remaining labels or gradient-based unlearning adapted from classification. They should do so at a fraction of the cost of replaying the whole annotation history.
The "never-included" reference is defined as follows. Let $\theta^{-k}$ be the model obtained by replaying the full annotation loop with rater $k$ absent. In replay, the other raters' edits are adjusted for the drafts they would have seen instead. The goal is a model $\hat\theta$ whose predictions $f_{\hat\theta}(x)$ track $f_{\theta^{-k}}(x)$. The required anchoring correction for a later rater $j$ on image $x$ is
$$ \tilde y_j(x) = y_j(x) - \alpha \,\big(d(x) - d^{-k}(x)\big), $$
where $y_j(x)$ is rater $j$'s recorded score, $d(x)$ is the draft they were shown, $d^{-k}(x)$ is the draft a model without $k$ would have shown, and $\alpha \in [0,1]$ is the anchoring coefficient: the fraction of a draft shift that raters carry into their final score. If $\alpha = 0$, retraining on the remaining labels is already exact and our third step is unnecessary. Measuring $\alpha$ is therefore the decisive empirical question.
Relation to Prior Work
Prior work plays three roles here: some motivates the gap, some supplies methods we reuse, and some constrains what can work.
Two classification unlearning methods motivate the gap and also serve as baselines. Learn to Unlearn for Deep Neural Networks (PGU) finds the gradient directions that matter for retained data and confines forgetting updates to the remaining directions. AdaProb optimizes substitute output probabilities toward uniform distributions on the forget set. Both define forgetting through categorical entropy or uniform outputs, so neither has a natural meaning for a continuous score. Our method needs no such surrogate, because the counterfactual target is known.
Machine Unlearning: Challenges in Data Quality and Access is the closest conceptual neighbor and also a constraint. It treats unlearning under noisy, partial, or label-free data, but only for classification. It also warns that forcing uniform outputs can over-unlearn and destroy generic representations. That warning is exactly why we avoid entropy objectives in a domain whose models depend on fine spatial cues, such as the distortion-sensitivity priors learned by DSMix.
Enhancing User-Centric Privacy Protection pairs user interaction with second-order unlearning on facial attributes. Its users control privacy noise, though; they do not produce the labels being deleted.
On the human-in-the-loop side, ActLocalizer supplies the propagation pattern. It spreads a few human corrections across thousands of segments through graph optimization, cutting annotation time by 78%. It also illustrates the problem: it keeps no record that would let a correction be withdrawn. The Dresden OCT dataset reports the more-than-15x speedup that makes draft-and-correct annotation attractive in the first place, which is why the anchoring channel will be common in practice.
For the base models we reuse MUSIQ, a transformer that scores images at native resolution, and DSMix-pretrained encoders.
Research Program
We will build a human-in-the-loop annotation system for image quality with three pieces of instrumentation.
First, every rating is stored with its rater, the draft score shown to that rater, and the model version that produced the draft.
Second, propagation is restricted to a form whose influence can be traced. Unlabeled images receive pseudo-labels as kernel-weighted combinations of corrected labels in encoder feature space. The weight each correction contributes is stored as an edge in a provenance graph. Deleting rater $k$ then means three things: recompute affected MOS values without $k$, recompute the affected pseudo-labels from their remaining edges, and recompute drafts and apply the anchoring correction to downstream edits.
Third, the model is updated by fine-tuning only on the affected images toward their new targets. A retained-data anchor term keeps predictions on untouched images fixed. Optionally, the update is confined to PGU's residual gradient directions. A pure-regression analogue of the subspace idea is part of the ablation.
Measuring $\alpha$ requires a controlled human study. Raters correct drafts that have been secretly shifted by known amounts, and we regress their final scores on the shift. We will estimate $\alpha$ overall and check whether it varies with image difficulty and rater fatigue. If $\alpha$ turns out to depend strongly on the image, the correction becomes per-image rather than a single global constant, and we will report that.
The systems arm is deliberately narrow. Deletion requires recomputing drafts and pseudo-labels for possibly thousands of native-resolution images while annotators keep working. We will test whether moving model weights and image tensors directly into remote GPU memory changes whether deletion fits inside the live session. Two mechanisms are in scope: GPUDirect RDMA, which moves network data straight into remote GPU memory without passing through host RAM or the CPU, as studied by Understanding the Benefits of Hardware-Accelerated Communication, and on-demand point-to-point parameter transfer, as in Revisiting Parameter Server in LLM Post-Training. We also expect the memory-pinning limits documented in NP-RDMA to bound how many annotators can be served at once. This arm is secondary, and it will be cut if commodity networking suffices.
Baselines
The naive reference point is deletion of raw ratings only. Rater $k$'s scores are removed, MOS values are recomputed, and the model is left as it is. This is what a database-level deletion does. It shows how much influence is stored in the weights rather than in the labels.
The gold reference is a full replay of the annotation loop without rater $k$. Previously recorded edits from other raters are adjusted by the measured $\alpha$, and the model is retrained from the same initialization and with the same hyperparameters. This defines $\theta^{-k}$. Because the edits are adjusted using measured anchoring rather than freshly collected, the reference is itself a model. We will validate it on a small subset where we re-collect real edits from raters shown $k$-free drafts.
The strongest practical competitor is retraining on the remaining labels. The model is retrained from scratch on everything not authored by $k$, with propagated pseudo-labels recomputed. It is cheap at KonIQ scale and exactly correct when $\alpha = 0$, so it is the baseline we most need to beat. It gets the same compute and tuning budget as our method's final model.
The strongest existing unlearning method is PGU adapted to regression. Its entropy objective is replaced by pushing predictions on the forget images away from $k$'s scores, it uses its subspace projection, and it gets the same affected-image set and step budget as ours. On CIFAR-10, PGU reports an 11.6x speedup over retraining (88s vs. 1021s), retained error within 0.01% of retraining, and membership-inference AUC of 0.5340 (Learn to Unlearn). No reported result was found on Lacuna for any unlearning method on an image quality benchmark. The data-quality unlearning label-agnostic method is run as a second competitor because it needs no labels.
Two ablations isolate the key ingredients. The first removes the anchoring correction, which amounts to our method with $\alpha = 0$. The second replaces the recomputed counterfactual targets with PGU-style push-away targets.
Beating the competitors means several things at once. Our predictions should be closer to $\theta^{-k}$, measured by mean absolute difference and by rank correlation between the two models' scores on held-out images, than those of retraining on remaining labels whenever the measured $\alpha$ is meaningfully above zero. We should match or beat PGU on that same distance. We should stay within 0.01 SRCC of the reference on the test split. And we should cost at least an order of magnitude less than full replay. For context, MUSIQ reports SRCC 0.916 on KonIQ-10k, and DSMix reports 0.646 on FLIVE with linear probing.
Why not just…?
Why not just retrain on the remaining labels? This is the serious threat. For a 10k-image dataset it is affordable, and it is exact if raters do not anchor on drafts. Our claim stands only if $\alpha$ is substantially non-zero. The drifted-draft human study settles this directly. If $\alpha$ is near zero, we will report that the third step is unnecessary, and the contribution shrinks to provenance-tracked propagation plus cheap targeted updates. The anchoring term would then be dropped.
Why not just give each rater a separate adapter and delete it? Removable adapters work when a person's influence stays inside their own module, as in the parameter-isolation designs surveyed by ERASE. In this loop, rater $k$'s corrections leave their module through pseudo-labels and through drafts shown to others. Adapter deletion can only ever remove the first channel. We will run it as an extra baseline to measure how much influence it leaves behind relative to $\theta^{-k}$.
Why not just use commodity TCP networking? Human review takes seconds, so moving drafts to the interface is unlikely to be bottlenecked by network latency. The only plausible bottleneck is the bulk recomputation of drafts and pseudo-labels during deletion. The cited serving study reports 70–160 ms latency savings and half the CPU load with GPUDirect, but only for forward-pass inference. We will time deletion end-to-end under TCP, RDMA, and GPUDirect. If the recomputation fits within a review interval under TCP, the hardware arm is dropped.
Proposed Experiments
The main study uses KonIQ-10k and SPAQ as seed pools, with MUSIQ and a DSMix-pretrained encoder as base models. It requires per-rater scores. Where public releases provide only aggregate scores, we will collect a new per-rater pool of roughly 2,000 KonIQ images. A run proceeds as follows: begin with a small labeled seed, run five draft-and-correct rounds with about 30 raters, and record all provenance. Then delete each rater in turn, including deliberately idiosyncratic ones such as raters who systematically penalize blur or noise.
Metrics fall into four groups. The first is distance of predictions to $\theta^{-k}$. The second is test SRCC and PLCC (Pearson correlation) against held-out human MOS. The third is a rater-signature test: on images rater $k$ scored but that were withheld from training, we correlate $k$'s deviation from the crowd with the model's residuals. The deleted model should show the same correlation as $\theta^{-k}$. This test replaces membership-inference AUC, which assumes discrete labels. The fourth is wall-clock cost.
Generalization is checked on CLIVE and FLIVE, and on the compressed-video setting of LEHA-CVQAD, where scarcity is most acute.
Two checks are decisive. The first is the measured $\alpha$ together with a prediction it implies: the gap between retraining on remaining labels and $\theta^{-k}$ should grow with $\alpha$ times the size of $k$'s draft influence. The second is whether recomputed counterfactual targets preserve ranking quality better than push-away targets when the affected images make up more than 10% of the data, which is where the over-unlearning warning predicts entropy-style methods will fail.
References
- Learn to Unlearn for Deep Neural Networks: Minimizing Unlearning Interference with Gradient Projection
- AdaProb: Efficient Machine Unlearning via Adaptive Probability
- Machine Unlearning: Challenges in Data Quality and Access
- DSMix: Distortion-Induced Sensitivity Map Based Pre-training for No-Reference Image Quality Assessment
- Enhancing User-Centric Privacy Protection: An Interactive Framework through Diffusion Models and Machine Unlearning
- Enhancing Single-Frame Supervision for Better Temporal Action Localization
- The Dresden in vivo OCT dataset for automatic middle ear segmentation
- MUSIQ: Multi-scale Image Quality Transformer
- Understanding the Benefits of Hardware-Accelerated Communication in Model-Serving Applications
- Revisiting Parameter Server in LLM Post-Training
- NP-RDMA: Using Commodity RDMA without Pinning Memory
- ERASE -- A Real-World Aligned Benchmark for Unlearning in Recommender Systems
- LEHA-CVQAD: Dataset To Enable Generalized Video Quality Assessment of Compression Artifacts