Degradation-Matched Exemplars for Judging Actions in Corrupted Video

Systems that score how well an athlete performed a dive, or how skillfully a surgeon passed a needle, usually work by comparison. They split the video into phases such as take-off, flight and entry, line each phase up against the matching phase of a reference video whose score is known, and predict the score from the difference. This works on clean broadcast footage. It should fail in a specific, predictable way when the query video is degraded by rain, haze, sensor noise, blur or compression while the reference is clean: part of the feature difference then measures the weather rather than the athlete, and the system marks the performance down for bad conditions. We propose a simple fix and a test of whether it is needed. Before comparing, estimate the degradation in each phase of the query, apply that same degradation to the matching phase of the reference, and only then compute the difference. Corruption that both videos share should largely cancel, so the difference tracks execution again.

The opportunity is that the field's default answers each solve a different problem. All-in-one restorers, single networks that remove several mixed corruptions at once, are tuned for pixel fidelity. Training the scorer on augmented degraded data makes it robust on average but does nothing about the mismatch between a degraded query and a clean reference. Matching the reference's degradation to the query's goes after that mismatch directly, needs no restorer at test time, and can be stacked on top of augmentation. The core experiment checks whether it removes a degradation-induced scoring bias that the other approaches leave in place.

Core Claim

Exemplar-based action quality assessment (AQA) predicts a query score as the exemplar score plus a learned function of the feature difference, $\hat{s}_q = s_e + R(\phi(q) - \phi(e))$. Here $\phi$ is the per-phase video feature, $R$ the regression head, and $s_e$ the known exemplar score. Suppose a degradation with parameters $d$ shifts features roughly as $\phi(D_d(x)) \approx \phi(x) + b(d) + r(x,d)$. The term $b(d)$ is the part that depends only on the degradation, and $r$ is the part where degradation and content interact. Comparing a degraded query with a clean exemplar then feeds $R$ an extra input $b(d_q)$, which shows up as a systematic score bias that grows with severity. Degrading the exemplar with the same $d_q$ cancels $b$ to first order and leaves only the interaction residual $r(q,d_q) - r(e,d_q)$.

The claim is narrow and falsifiable. Under compound, phase-varying degradation, per-phase degradation matching reduces the correlation between score error and degradation severity, and it improves rank correlation with judges. It should beat both restoration-then-scoring and augmentation-trained scoring, and it should add to augmentation rather than duplicate it. If an augmentation-trained scorer already shows no severity-dependent bias, the claim is wrong, and we will report that.

Where This Sits

Two families of procedure-aware scorers supply the method we modify. FineDiving segments each dive into take-off, flight and entry with a temporal convolution block, then aligns each query step to the same step of a reference via procedure-aware cross-attention before regressing a score offset. Temporal Parsing Transformer finds phases without frame labels, using learnable queries kept in order by a ranking loss and kept compact by a sparsity loss, and scores by part-aware contrastive regression against exemplars. Both quietly assume the query and the exemplar look alike, because both were evaluated only on clean footage. We keep their phase parsing and contrastive regression and change what the exemplar looks like at comparison time.

The temporal parsing paper also constrains the design. Its authors found that a standard Transformer encoder hurt phase parsing because global attention smooths features across neighboring frames. That is a warning about cascades, since the video restorers we would otherwise put in front of the scorer depend on multi-frame aggregation. UniWRV fuses adjacent frames along dynamically chosen temporal paths to remove hybrid weather. RONIN runs a streaming recurrent U-Net steered by language-derived degradation prompts. Neither has been evaluated on downstream temporal tasks. We treat them as the strongest competing route, not as components.

The degradation-estimation side borrows from AirNet, which learns a label-free degradation embedding by contrasting patches. It assumes every patch of an image shares one degradation and does not model motion blur. We need the opposite: an estimate that is explicit and parametric, so it can be re-rendered onto another video, and that varies over time, so it can follow phase-specific corruption such as splash at entry. The physical observation in Deep Joint Rain Detection and Removal from a Single Image, that heavy rain produces both streaks and a veiling mist, motivates rendering compound effects jointly rather than adding independent noise.

Older structured localizers, Structured Segment Networks and Structured Maximal Sums, show that start/course/end structure helps localization and that frame-wise noise collapses short phases. They motivate measuring phase-boundary accuracy alongside the score. They are not our target task.

Research Program

The first deliverable is a benchmark, because none exists. We will build degraded versions of FineDiving and MTL-AQA. Each clip is corrupted by a compound, time-varying synthesizer covering rain streaks with veiling haze, snow, low light with shot noise, motion blur and compression, at controlled severities. The pool of conditions is modeled on the hybrid conditions of HWVideo and the time-varying protocols RONIN uses. A phase-localized variant concentrates corruption in one phase, for example heavy spray-like occlusion and blur in entry only, using the FineDiving sub-action boundaries. Every degraded clip keeps its parameters, which gives us oracle degradation labels and lets us generate pairs of the same dive at two severities.

The second deliverable is the matching mechanism. A light per-phase degradation estimator, a small 3D convolutional network, regresses the synthesizer's parameters from short windows of the query. It is trained only on synthetic corruptions of non-test clips. At scoring time the phase parser runs on the degraded query. The estimator predicts parameters for each parsed phase, the synthesizer renders those parameters onto the exemplar's corresponding phase, and contrastive regression runs on the matched pair. The scorer itself, I3D features with TSA or temporal parsing heads, is trained with matched pairs: during training, both members of each pair receive the same sampled degradation. Because rendering is parametric, the method's success depends on how well the estimator generalizes. We measure that gap directly by comparing oracle and estimated parameters.

The third deliverable is a diagnostic: degradation-induced score bias. For each method we regress score error on query severity while controlling for the true score. We also report the score difference for the same dive at two severities, which should be zero for an ideal scorer. This tests the mechanism, not just the headline correlation.

Baselines

The clean reference point is each scorer on undegraded test clips. The file reports TSA on FineDiving at Spearman 0.9203 with relative $\ell_2$ of 0.3420 and sub-action segmentation of 82.51% [email protected] (FineDiving), and the temporal parsing transformer on MTL-AQA at Spearman 0.9607 (Temporal Parsing Transformer). These are ceilings, not targets. No reported result was found on Lacuna for any AQA method on degraded video.

The do-nothing baseline runs the clean-trained TSA and temporal parsing models on the degraded test sets with clean exemplars. It measures how large the problem is.

The strongest competitor is the augmentation-trained scorer. It is the same architecture trained on the same compound synthesizer, with query and exemplar degraded independently at random, and it receives the same number of epochs and the same hyperparameter search budget as our method. This is the competitor a practitioner would actually ship. It differs from our method only in whether the exemplar's degradation matches the query's, which makes it the cleanest comparison.

The restoration cascade first restores degraded queries with UniWRV and with RONIN, retrained on our synthesizer's distribution, then scores them with the clean-trained scorer. A second variant fine-tunes the scorer on restored outputs. Their published restoration figures, UniWRV's 3.31 dB PSNR gain over RVRT on HWVideo (UniWRV) and RONIN's 32.36 dB average PSNR on its 3D benchmark (RONIN), show they are strong restorers. They say nothing about scoring. We also report wall-clock cost, since a restorer adds a full video network in front of I3D.

Three ablations isolate the key ingredient. The first degrades the exemplar with a random, mismatched degradation of equal severity, which separates matching from simply degrading the exemplar. The second matches one global degradation per clip instead of one per phase. The third substitutes oracle parameters for estimated ones to bound how much depends on estimator quality.

We count it as beating the strongest competitor if the matched method improves Spearman over the augmentation-trained scorer on both degraded benchmarks across three seeds, and cuts the severity-bias slope by at least half. Combining matching with augmentation should also beat augmentation alone.

Why not just…?

Why not just restore the video first and score the clean-looking output? Restorers are trained for pixel fidelity, and the temporal parsing paper shows that temporal smoothing destroys the phase boundaries that scorers rely on. Restorers also hallucinate content where the subject is occluded, and that content can look like an execution error. The evidence is indirect, though, and a strong restorer may well recover most of the clean score. The restoration-cascade baseline, together with phase-boundary AIoU on restored clips, settles this. If the cascade matches us on accuracy, our remaining advantage is cost, and we will say so.

Why not just train the scorer on heavily augmented degraded data? This is the real threat. A large enough scorer might learn features in which $b(d)$ is already near zero, and matching would then add nothing. The counterpoint is that perfect invariance tends to discard the fine detail, such as toe point or hand position, that separates close scores. Matching lets the features keep that detail and cancels the degradation by subtraction instead. The bias diagnostic decides between these readings. If the augmentation-trained scorer shows a flat severity slope and equal accuracy, the idea is unnecessary.

Why not just align degraded and clean features with domain adaptation? Adversarial or contrastive domain alignment matches feature statistics across whole clips. Phase-wise scoring needs take-off compared with take-off, and corruption is often concentrated in one phase. Global alignment cannot follow corruption that changes from phase to phase. The phase-localized benchmark, together with the global-versus-per-phase matching ablation, tests this. If global treatment works as well there, the per-phase design is not justified.

Proposed Experiments

The main table reports Spearman correlation and relative $\ell_2$ on degraded FineDiving and degraded MTL-AQA, at three severity levels and in the phase-localized setting, for every baseline and ablation, averaged over three seeds. A second table reports sub-action segmentation [email protected] under degradation. It separates failures of phase parsing from failures of comparison, since matching cannot help if the phases are parsed wrongly.

Two decisive checks follow. The first is the bias analysis: the slope of score error against severity, and the same-dive two-severity score gap. Our account predicts that the do-nothing and augmentation baselines show a negative slope, meaning degraded dives are scored lower, and that matching flattens it. A flat slope for augmentation alone falsifies the motivation. The second is a transfer test. The estimator and scorer are trained on one family of corruptions, say weather, and evaluated on held-out compound mixtures that add compression and low light, with estimated rather than oracle parameters. The method stands only if most of the oracle-matching gain survives estimation error on unseen mixtures. A large gap would mean the bottleneck is degradation estimation, which would redirect future work toward better degradation estimation for video.