Action quality assessment (AQA) asks a video model to say how well a movement was performed, not what the movement was. Typical inputs are an Olympic dive, a surgical suture, or a gymnastics routine. Progress in the field is reported as small gains in rank correlation with expert scores. Reported Spearman correlations on MTL-AQA, an Olympic diving benchmark, are 0.951 for one method and 0.9545 for its successor. Those numbers are computed on a few hundred test videos. They come from a single training run. For the strongest methods, they also depend on a random draw of "exemplar" reference videos that the model compares against at test time. Nobody has measured how much of such a gap would survive a different seed, a different exemplar draw, a different test sample, or a different panel of judges.
We propose to measure this, and to remove the part that can be removed. The measurement is a nested resampling study that splits the variance of a reported correlation difference into four sources: training seed, exemplar draw, test-video sample, and judge panel. The method targets the one source unique to AQA, which is the random choice of exemplars. We replace random multi-exemplar voting with exemplars stratified by score, and we weight each exemplar's vote by how reliable that score distance proved on validation data. The decomposition is the honest error bar for AQA results. The exemplar fix tests whether that error bar can be narrowed for free.
Core Claim
Our claim is that AQA comparisons carry variance that a standard test-set bootstrap cannot see. The strongest methods are contrastive: they predict a score difference against reference videos. As a result, their reported number is itself a random variable over exemplar draws. Training seeds add more spread, and the labels are noisy averages of a few judges. We expect two findings. First, a naive bootstrap will under-cover, meaning its nominal 90% intervals will contain the result of a fresh replicate less than 90% of the time. Second, at the 20-sample cohort sizes used in clinical surgical assessment, ranking flips between published methods will be common. We further claim that score-stratified exemplar selection removes most of the exemplar component without lowering mean correlation. If exemplar and seed variance turn out to be negligible, the result is still useful: a plain paired bootstrap would then be sufficient for AQA, and we would report that.
Prior Work and Its Role
Two prior works supply the motivation from the evaluation side. Two Sides of Meta-Learning Evaluation showed that few-shot classification benchmarks built on about 20 test classes flip the order of two methods up to 15% of the time. On those benchmarks, a true 0.5% margin is exaggerated 25% of the time. That paper also showed that validation performance correlates poorly with test performance. Its analysis covers classification accuracy, not rank correlations of continuous scores, and it has no analogue of exemplar sampling. FlashEval found that random 50-prompt subsets rank text-to-image models much worse than searched subsets: a random subset needs about 500 items to reach the Kendall's tau of about 0.8–0.9 that a searched 50-item subset achieves. This constrains our design. It warns that small random draws are noisy, and it suggests that structured selection (in our case, structured exemplar selection) can outperform random selection at equal cost.
Three AQA works supply the methods whose variance we study. Group-aware Contrastive Regression introduced contrastive regression (CoRe): it predicts the difference between a query's score and an exemplar's known score, then averages over several exemplars at inference. The authors leave exemplar selection heuristic and name it as an open problem. FineDiving aligns each dive's take-off, flight, and entry steps against exemplars of the same dive type, reaching a Spearman correlation of 0.9203 versus 0.9061 for CoRe. Some dive types and difficulty levels have very few videos, which narrows the exemplar pool. Uncertainty-Driven AQA models judge disagreement with a conditional variational auto-encoder, a generator that samples plausible scores from a learned distribution. It also down-weights ambiguous videos during training. It treats disagreement as a training signal and never asks what judge noise does to the test metric. We use its per-judge view of the labels for the opposite purpose: resampling judges to measure label-induced variance at evaluation time.
Towards Unified Surgical Skill Assessment reports a Spearman correlation of 0.565 on 20 real gastrectomies with no interval. That cohort size is the regime in which we expect comparisons to be least trustworthy. SimCalibration reduces benchmarking variance on tiny tabular cohorts by simulating data. We do not adopt that route, because a credible generative model of surgical video does not exist, and the simulation would add approximation bias that the paper itself acknowledges.
Research Program
The first part is the variance decomposition. We train each studied method under several seeds. For each trained model we draw several exemplar sets. Each resulting prediction set is scored on bootstrap resamples of the test videos. On MTL-AQA, which provides individual judge scores, each resample also recomputes the official aggregate score from a resampled judge panel. The quantity of interest is the paired difference in Spearman correlation between two methods. We fit a crossed random-effects model to these differences. That model yields the share of variance contributed by each source and a predictive interval for what a fresh replicate would report.
We then subsample the test set to sizes from 20 up to the full set and extrapolate. From this we derive two quantities per benchmark: the minimum detectable correlation difference, and the flip probability for each published pair of methods. This turns the clinical cohort size into a quantitative warning rather than an anecdote.
The second part is the variance-reduction method. CoRe-style inference averages $\hat s = \frac{1}{K}\sum_{k} (s_{E_k} + \hat\Delta_k)$. Here $s_{E_k}$ is the known score of exemplar $k$, $\hat\Delta_k$ is the predicted difference between query and exemplar, and the $K$ exemplars are drawn at random from eligible training videos. Our variant instead partitions the eligible pool, which keeps the same dive type where required, into $K$ score quantiles and draws one exemplar per quantile. It then weights each term by the inverse of its validation error at that query–exemplar score distance. The weights are fitted once on validation data, so inference cost does not change.
Baselines
The naive reference point is the current reporting practice: one training run, one exemplar draw, one number. We reproduce the published configurations of CoRe with GART, TSA, and Uncertainty-Driven AQA from their released settings. For each, we report our single-run number next to the paper's number, to show where reproduction already deviates.
The strongest statistical competitor is a paired bootstrap over test videos applied to that single run. This is what a careful reviewer would ask authors to add. It is the main threat, because it is cheap and standard. We run it on exactly the same prediction files used in our decomposition. It beats nothing unless it fails at the job it claims to do. The fair test of both is coverage. On a held-out half of the test pool, we retrain with fresh seeds, draw fresh exemplars, and check how often each method's 90% interval contains the new paired difference. Our protocol wins if its coverage lands within three points of nominal while the bootstrap's falls clearly below.
The strongest modeling competitor is random multi-exemplar voting as in CoRe. On FineDiving its reported Spearman correlation is 0.9061 (FineDiving), and TSA's is 0.9203. Our stratified variant runs with the same backbone, checkpoint, and number of exemplars. Beating it means cutting the exemplar-induced standard deviation of Spearman correlation at least in half, with mean correlation no lower than our reproduced random-voting mean.
Two ablations isolate the key ingredient: stratification without weighting, and weighting without stratification. A further control is a direct absolute-regression model, which has no exemplar component. It shows how large the remaining variance is when exemplar variance is absent by construction.
Why not just…?
Why not just report a paired bootstrap over test videos? It captures only one of four sources. If the decomposition shows that seed and exemplar variance are small, this alternative wins, and we will say so. The coverage experiment above decides it directly rather than by argument. Evidence from few-shot learning (Two Sides of Meta-Learning Evaluation) shows that the sample of the evaluation pool alone can flip conclusions. That is a reason to expect the other sources to matter on top of test-sample variance, but the AQA case is untested.
Why not just average over more random exemplars? More exemplars do shrink variance, at a linear cost in inference. For long videos and small same-type pools, more exemplars may not even exist. We plot exemplar-induced variance against $K$ for random and stratified voting. If random voting at $2K$ matches stratified voting at $K$, the stratified method saves real compute. If random voting at $K$ already matches it, our method is unnecessary. The FlashEval result, where structured 50-item subsets matched random 500-item ones, suggests the gap can be large. That result comes from a different task, however.
Why not just drop exemplars and regress absolute scores? That removes exemplar variance, but it costs accuracy. CoRe's own comparison shows absolute regression at 0.70 versus 0.85 on JIGSAWS and 0.810 versus 0.840 on AQA-7 (Group-aware Contrastive Regression). Our absolute-regression control will also show whether that accuracy gap exceeds the total noise we measure. If it does not, the contrastive advantage itself is not established, which would be a substantive finding.
Why not just drop low-agreement videos from the test set? Removing ambiguous videos shrinks the evaluation pool. When does dough become a bagel? argues that shrinking a pool this way makes progress harder to separate from fluctuation. On MTL-AQA we compute the flip probability on the full test set and on a high-agreement subset to measure this trade-off directly.
Proposed Experiments and Decisive Checks
We will use MTL-AQA for the judge-panel component and FineDiving for exemplar variance within sparse dive types. JIGSAWS serves as the small surgical benchmark. Twenty-video subsamples of FineDiving and JIGSAWS stand in for the clinical gastrectomy cohort, which we treat as the target regime rather than as data we have.
Each method gets five seeds, ten exemplar draws per checkpoint, and one thousand test bootstraps. We also record the gap between checkpoints selected on validation data and the best checkpoint in hindsight, following Two Sides of Meta-Learning Evaluation.
Three checks decide the proposal. The first is whether the gap between CoRe with GART and Uncertainty-Driven AQA on MTL-AQA (0.951 versus 0.9545) exceeds our minimum detectable difference. The second is whether naive bootstrap intervals under-cover on fresh replicates. The third is whether stratified exemplars halve exemplar-induced spread at equal $K$. A "no" on the first would mean recent MTL-AQA leaderboard movement is not distinguishable from noise. A "no" on the second would vindicate the simple bootstrap. A "no" on the third would mean exemplar variance is not removable by selection alone.
References
- Two Sides of Meta-Learning Evaluation: In vs. Out of Distribution
- FlashEval: Towards Fast and Accurate Evaluation of Text-to-image Diffusion Generative Models
- Group-aware Contrastive Regression for Action Quality Assessment
- FineDiving: A Fine-grained Dataset for Procedure-aware Action Quality Assessment
- Uncertainty-Driven Action Quality Assessment
- Towards Unified Surgical Skill Assessment
- Meta simulation approach for evaluating machine learning method selection in data limited settings
- When does dough become a bagel? Analyzing the remaining mistakes on ImageNet