Learning Where Video Quality Breaks from a Single Score

When people rate a video's quality, they give one number for the whole clip. That number is usually driven by a few bad moments and regions: a second of heavy compression blocking, a smeared face during camera shake, a patch of sensor noise in a dark corner. Blind video quality assessment, meaning prediction of human quality ratings with no clean reference video to compare against, has become accurate at predicting that one number. It still cannot reliably say where the quality broke unless someone pays for dense local labels. The largest such effort, Patch-VQ, collected 5.5 million human ratings on 116,433 space-time patches to learn local quality. We propose to treat each video as a bag of local space-time pieces and learn per-piece quality from the clip-level score alone. This is the setting of multiple instance learning (MIL): the label belongs to the collection, and the model must work out which members caused it. We would then check the learned local scores against Patch-VQ's patch labels, which are never used in training.

The second half of the idea addresses a known failure. Quality models confuse intended softness, such as a blurred background behind a sharp subject or deliberate film grain, with capture defects. They also overrate synthetic overlays and gaming graphics (DOVER, RQ-VQA). This is a small instance of the semantic gap, the mismatch between low-level pixel statistics and what the content means to a viewer. The same local blur is a defect on a face and a stylistic choice on a background. Our model therefore gives each piece two frozen descriptions. One comes from a distortion-sensitive encoder that sees raw pixels. The other comes from a semantic encoder that sees the piece in its scene context. The aggregator uses the semantic description only to decide how much each piece's technical defect should count toward the final score.

Core Claim

Clip-level mean opinion scores (MOS, the averaged human rating) contain enough signal to learn local quality maps that correlate with human patch ratings. This holds if the bag aggregator encodes two things. The first is the asymmetric way humans pool quality: bad moments dominate, and recovery after an impairment is slow. The second is semantic context about whether a local degradation matters. Standard attention MIL has neither. It is permutation-invariant, so a glitch at second one and at second nine count the same, and its weights tend to spread thinly over mostly clean pieces. Existing quality models also lack an instance-level decision. FAST-VQA regresses a score per patch and then simply averages. DOVER fuses its aesthetic and technical branches with one fixed global weighting, derived from the roughly 57/43 technical/aesthetic split it measured. ModularBVQA combines its rectifiers through fixed operations. What is new here is an order-aware, semantically gated MIL regressor over frozen features, together with a test of weak localization against held-out dense human labels.

Research Program

Instances and features. Each clip is split into space-time cubes. For cube $i$ we compute two frozen features, following the lesson of Could Giant Pre-trained Image Models Extract Universal Representations? that frozen backbones with well-placed heads regularize well when labels are scarce.

The technical token $t_i$ comes from a FAST-VQA-style fragment encoder on raw-resolution mini-patches. Raw resolution preserves the high-frequency detail that resizing destroys. The semantic token $c_i$ comes from a frozen CLIP image encoder applied to a downsampled crop around the cube that includes surrounding context. The research file notes that semantic encoders are largely invariant to blur and noise, which is exactly why $c_i$ is used only for routing and never to score quality directly. A mean pool of CLIP features could not help separate a blurred face from a blurred background; that is the point of routing through context.

Scoring and spatial pooling. A small head scores each cube as $s_i = f(t_i)$. Within each time step $\tau$ we pool spatially with weights

$$a_i = \mathrm{softmax}_i\big(g(c_i, t_i) - \beta s_i\big),$$

where $g$ is a learned relevance function of semantics and distortion. The learned $\beta \ge 0$ tilts attention toward low-scoring cubes, giving a soft version of the "worst regions dominate" effect. This produces a per-step score $q_\tau = \sum_i a_i s_i$.

Temporal pooling. Across time we replace permutation-invariant pooling with a differentiable hysteresis term, in the spirit of the psychovisual models compared in A Comparative Evaluation of Temporal Pooling Methods:

$$Q = \sum_\tau w_\tau \Big(\lambda\, q_\tau + (1-\lambda)\, \mathrm{softmin}_{\tau' \in [\tau-K,\tau]} q_{\tau'}\Big).$$

Here $Q$ is the predicted clip MOS, $w_\tau$ are learned temporal weights, $\lambda \in [0,1]$ is learned, and $K$ is a memory window. The window encodes slow recovery after a dip.

Training and analysis. Training uses only clip MOS, with a correlation-plus-regression loss. After training, $s_i$ is the quality map and $a_i$ is the "what counted" map. We analyze whether $g$ learns to down-weight defocused backgrounds and overlay text, and whether the learned $\lambda$ and $K$ resemble the hysteresis parameters reported in psychophysics.

Baselines

The naive reference is a frozen CLIP-plus-fragment feature averaged uniformly over all cubes and frames, followed by a small multilayer perceptron. It uses exactly our features, so any gain comes from aggregation alone.

The closest methodological competitor is FAST-VQA's own intra-patch regression with spatial averaging. It is essentially mean-pooling MIL, which makes it the precise test of whether our pooling adds anything. We rerun it on our frozen features and also report its published configuration.

The pooling-strength competitor is a worst-k% percentile pool of per-cube scores, the best simple rule in the temporal pooling comparison. That paper reports roughly a 0.03 SRCC gain over mean pooling on LIVE-VQC. SRCC, Spearman rank correlation with human scores, is the main metric throughout.

The dense-supervision reference is Patch-VQ. It reports SRCC 0.827 on LSVQ, and it is the only baseline that saw patch labels. We found no reported patch-level localization number for it on Lacuna, so we will retrain its patch head to obtain that figure.

The strongest accuracy competitor is RQ-VQA, which concatenates several frozen expert features into a multilayer perceptron. It reports SRCC 0.926 and PLCC 0.924 on the KVQ test set, where PLCC is Pearson linear correlation. We also include DOVER, for which the file reports no absolute LSVQ value, only a gain of "up to 2.0%" PLCC. All methods share LSVQ training splits, feature extractors where applicable, and an equal hyperparameter search budget.

Three ablations isolate the proposal's ingredients:

  • removing $c_i$ from $g$, which leaves pooling with no semantic gate;
  • setting $\beta=0$, which removes the worst-case tilt;
  • setting $\lambda=1$, which removes hysteresis.

What counts as beating the baselines. On clip scores, we must match or exceed the percentile pool and FAST-VQA averaging on LSVQtest, LSVQ1080p and cross-dataset KoNViD-1k and LIVE-VQC. On held-out v-patch labels, we must clearly beat every weakly supervised baseline's localization. Coming within reach of Patch-VQ's densely supervised patch head would be the headline result.

Why not just…?

Why not just use worst-k% percentile pooling over patch scores? This is the real threat for clip-level accuracy. A fixed percentile rule may already capture most of what $\beta$ and the hysteresis term buy. We treat it as the primary comparison. If it ties us on clip MOS, the contribution must rest on two other tests. The first is localization: a percentile rule gives each patch an equal vote and has no notion of relevance. The second is the semantically confounded subsets, where a percentile rule should penalize bokeh backgrounds and fail.

Why not just fine-tune a video backbone end to end? Frozen features regularize better when labels are scarce, per the SwinV2-G study. End-to-end training at full resolution is also costly; FAST-VQA reports a 99.5% FLOP reduction relative to a full-resolution Swin. Most importantly, fine-tuning gives no instance-level readout to test against the patch labels. We will still run a fine-tuned FAST-VQA as an accuracy reference.

Why not just concatenate strong frozen experts, as RQ-VQA does? RQ-VQA wins on KVQ. Its authors, however, report systematic overestimation on synthetic content and text overlays, and a runtime of 5.3 seconds per 8-second 1080p clip. Concatenation pools the whole frame and cannot decide that an overlay region should be judged differently. The overlay and gaming subsets test this directly.

Why not just use DOVER's aesthetic/technical split? DOVER already separates semantics from technical quality, but with one global weighting and fragments that break up objects. Our ablation that removes $c_i$ from $g$ is the fair version of that question: if semantic gating per instance adds nothing over a global split, the ablation will tie the full model.

Proposed Experiments

Localization. We train on the LSVQ training split using clip MOS only. We then correlate learned cube scores, pooled to v-patch extents, with the held-out v-patch human ratings. This is the decisive test of the core claim, and it is compared against the retrained Patch-VQ patch head.

Clip-level accuracy. We report SRCC and PLCC on LSVQtest, LSVQ1080p and KVQ. We report cross-dataset transfer to KoNViD-1k and LIVE-VQC without retraining.

Semantic confounds. We use DIVIDE-3k's separate technical and aesthetic scores to check that the technical score tracks $s_i$ while the gate absorbs aesthetic variation. We also build two small diagnostic subsets from LSVQ and KVQ with manual verification: one of clips with shallow depth of field, and one of clips with text or graphic overlays. We add LIVE-YT-Gaming, for which no reported numbers were found on Lacuna. The prediction is that the full model's advantage over percentile pooling and over RQ-VQA-style concatenation is largest on these subsets.

Temporal ordering. We create a controlled temporal test by splicing a short compression burst at the start, middle or end of otherwise clean clips. A permutation-invariant pool must score all three placements identically. Our model should not, and its ordering can be compared qualitatively with the hysteresis patterns in the pooling comparison study. Runtime is reported against DOVER's 47 ms and FAST-VQA's 0.04 s per clip.