Systems that score athletic or surgical performance from video now work in two steps. First a segmenter cuts the video into phases, such as take-off, flight and entry for a dive. Then a scorer compares each phase against a reference performance and outputs a number. Nobody knows how far the final score can move when the segmenter places a phase boundary a few frames off. This matters because misplaced boundaries are the main way temporal detectors fail. We propose to build the scorer so that its sensitivity to each boundary has a known, enforced limit. We then attach a calibrated error radius to the segmenter. Together these give every scored video a guaranteed interval, of the form "the score is 78.4 ± 1.1, and 0.8 of that width comes from the entry boundary."
The idea borrows from formal verification. There, "compositional error bounding" means proving that when approximate components are wired together, the error of the whole is at most a sum of per-component errors. The reason is that each connecting operation is non-expansive, meaning it cannot amplify differences between its inputs. We apply that discipline to temporal action decomposition, the practice of splitting an action into ordered sub-phases. The result is a scorer whose total error splits into a boundary term and a scoring term. Each term can be measured, calibrated and swapped independently.
Core Claim
Procedure-aware quality scorers can be made Lipschitz in their phase boundaries at almost no cost in ranking accuracy. A function is Lipschitz with constant $L$ if moving its input by $\delta$ moves its output by at most $L\delta$. Write $\hat b$ for the predicted boundary times and $b^*$ for the true ones. Write $\hat y(b)$ for the score computed from boundaries $b$, and $y^*$ for the judges' score. The error then splits as
$$|\hat y(\hat b) - y^*| \;\le\; \sum_{k=1}^{K} L_k\,|\hat b_k - b^*_k| \;+\; |\hat y(b^*) - y^*|,$$
where $L_k$ is the scorer's enforced sensitivity to boundary $k$ and $K$ is the number of boundaries. The first term is the segmenter's fault and the second is the scorer's fault.
We calibrate each term separately on held-out data, using split conformal calibration. This procedure turns held-out errors into radii that hold with a chosen probability without any assumption on the error distribution. It gives a boundary radius $r_k$ with joint coverage $1-\alpha$ and a scorer residual radius $q$ with coverage $1-\beta$. A union bound then yields a per-video score interval $\hat y(\hat b) \pm (\sum_k L_k r_k + q)$ with coverage at least $1-\alpha-\beta$.
The novelty is not a better ranking score. It is a decomposed, per-video, per-phase guarantee. That guarantee survives replacing the segmenter, which needs only recalibrating $r_k$, and it says which phase is responsible for the uncertainty.
Why This Gap Is Real
The motivating evidence comes from Diagnosing Error in Temporal Action Detectors. It splits detector false positives into five types. It finds that fixing localization error (correct class, wrong timing) is worth up to 10.9% mAP on ActivityNet, more than fixing classification errors. It also finds that failures are driven by missing temporal context rather than by annotators disagreeing about boundaries. Boundary error is therefore the dominant error source, and it is not merely label noise.
The phase-based scorers that consume these boundaries have no defined sensitivity to them. FineDiving segments a dive into consecutive steps, cross-attends each step to an exemplar's matching step, and regresses a score difference. FineParser adds human-centric spatial masks and a separate temporal parser. FineCausal models how early-phase errors propagate to later phases through learned "temporal causal attention." All three pass predicted boundaries into dot-product cross-attention, which is not globally Lipschitz. None states how score error scales with boundary error. FineCausal's causal story in particular is carried entirely by unconstrained attention weights.
NS-AQA constrains our design. It scores dives with rulebook microprograms on top of neural perception and reaches strong segmentation ([email protected] of 77.17% versus 34.31% for neural models) along with expert preference. Its hard thresholds, however, are discontinuous: an arbitrarily small boundary or pose shift can flip a deduction on or off. This is exactly the kind of interface a compositional bound forbids, so our scorer must be continuous in the boundaries by construction.
The method draws on three sources. Behavioral metrics and non-expansive composition come from Desharnais et al., Interface simulation distances, and the "independent implementability" results for weighted specifications of Bauer et al.. Bauer et al. show that components can be implemented separately so long as their local errors sum within budget, which is the property we want between segmenter and scorer.
Two learned-system precedents show such bounds can coexist with neural components. Port-Hamiltonian neural networks bound the error of a composite dynamical system by the errors of separately learned subsystems, by hard-coding structure rather than adding loss terms. The robust neural probabilistic circuits work proves that a concept-bottleneck classifier's downstream reasoning stage adds no fragility beyond its concept extractor's. We aim for the analogous statement that the scorer amplifies boundary error by at most $L_k$.
Rischel and Weichwald prove that abstraction error is sub-additive under composition, $\text{error}(fg) \le \text{error}(f)+\text{error}(g)$, when it is measured with a true metric. Their framework is restricted to finite discrete models. We take only its moral, that the chosen error measure must satisfy a triangle inequality for errors to add. We do not claim to extend their category to video.
Research Program
The scorer has three parts.
Boundary-differentiable phase pooling. Each phase feature is a weighted average of frame features. The weights come from a window whose edges are smooth ramps of width $\sigma$ centered on the boundaries. The derivative of the pooled feature with respect to boundary $k$ is then bounded by roughly $2\max_t\|x_t\|/(\sigma \cdot \ell_k)$, where $x_t$ is the feature of frame $t$ and $\ell_k$ is the phase length. This form follows from differentiating a normalized ramp-weighted average. We bound $\|x_t\|$ by normalizing frame features, and we enforce a minimum phase length.
Lipschitz exemplar comparison. Dot-product cross-attention is replaced with a comparison whose Lipschitz constant is known. One candidate is L2-distance attention with spectrally normalized projections. A simpler candidate compares phase features against exemplars and passes the result through a spectrally normalized MLP; we test whether this simpler version loses anything. The product of these per-module constants gives a certified $L_k$.
Calibration layer. Boundary radii $r_k$ are calibrated on a held-out split using the maximum over $k$ of normalized boundary errors, which gives joint coverage. Scorer residuals are calibrated on the same split, but with the scorer fed ground-truth boundaries.
The analysis has two parts. First, we measure how loose the certified $L_k$ is compared with the empirical worst-case sensitivity, found by grid search over boundary shifts per video. Second, we study the trade-off that $\sigma$ controls. A wide ramp lowers $L_k$ but blurs the brief, decisive moments, such as entry, that motivated phase decomposition in the first place.
Baselines
The naive reference is a constant-width interval. It takes an unmodified phase-based scorer and wraps its final score in a split-conformal interval of fixed width, calibrated on the same held-out split. It achieves coverage trivially and tells us how much a per-video, per-phase interval adds.
The strongest accuracy competitor is FineCausal. It reports Spearman ρ of 0.9447 and relative ℓ2 distance of 0.2338 on FineDiving-HM, just above FineParser at ρ 0.9435. On the original FineDiving, the TSA pipeline reports ρ 0.9203 and relative ℓ2 0.3420 (FineDiving). We will retrain FineCausal (or FineParser, if FineCausal's code is unavailable) on our splits and feature backbone, with the same tuning budget we give our method. The aim is accuracy parity, not beating it: we count our method as viable if its ρ stays within 0.01 of the retrained baseline. For certified score intervals on these benchmarks, no reported result was found on Lacuna, so the interval comparison is against baselines we construct ourselves.
The strongest uncertainty competitor is adaptive conformal on the final score. It trains a second network to predict each video's expected score error and scales a conformal interval by that prediction. Our method beats it if, at matched coverage, it gives narrower average intervals, a higher correlation between interval width and realized error, or both.
The key ablation keeps our architecture but removes the Lipschitz constraints (unnormalized projections, hard boundary cuts) while keeping the calibration layer. Without constraints, $L_k$ can only be estimated empirically, so this ablation shows whether the guarantee actually depends on the enforced structure.
Why not just…?
Why not just train the existing scorer with boundary-jitter augmentation? This is the most serious threat to any robustness claim. Augmentation may well make FineCausal empirically insensitive to small shifts. It gives no bound, however, and robustness learned at one jitter scale need not transfer to another, such as the errors of a weaker segmenter. We will train the strongest baseline with jitter matched to the calibration-set boundary error distribution and compare. If augmented FineCausal matches our measured sensitivity, we will concede that empirical robustness is cheap. The contribution then narrows to certification and attribution, which augmentation cannot provide.
Why not just apply conformal prediction directly to the final score? This also gives valid coverage, and with a learned difficulty model it can adapt to each video. It cannot, however, attribute width to phases. It must also be fully recalibrated whenever the segmenter changes, because the score residuals change. The segmenter-swap experiment below settles this. Our method recalibrates only $r_k$, using boundary labels, which are cheaper than score labels, while the direct method recalibrates on scored videos. If the adaptive direct method gives tighter intervals and the swap advantage is small, that is a real negative result and we will report it.
Why not just use a better segmenter, given that FineParser already reaches 0.9946 [email protected]? High overlap at a 0.5 threshold still allows several frames of error. That margin matters for brief, decisive moments like water entry, and FineCausal's [email protected] of 0.9453 shows boundaries do slip at stricter thresholds. Better segmenters also shrink $r_k$ inside our interval rather than replacing it. Our perturbation study measures directly how much score error frame-level boundary shifts produce. If that effect is negligible even for unconstrained scorers, the bound is unnecessary, and we would say so.
Proposed Experiments
We work on FineDiving and FineDiving-HM because both provide sub-action boundary annotations and official scores. We use their standard splits, carving a calibration split out of training.
The perturbation study shifts each ground-truth boundary by ±1 to ±8 frames. It records the score change for the retrained FineCausal or FineParser baseline, the jitter-augmented baseline, the ablation, and our scorer. It reports measured sensitivity alongside our certified $L_k$.
The interval study reports four quantities: empirical coverage at $1-\alpha-\beta = 0.9$, mean interval width, correlation between interval width and absolute error, and per-phase attribution of width. It compares our interval against the constant-width and adaptive conformal baselines.
The swap study calibrates the scorer once. It then pairs the scorer with two segmenters of clearly different quality, a TSA-style segmenter and a FineParser-style temporal parser, recalibrating only $r_k$. It checks whether coverage holds and whether width tracks segmenter quality.
Accuracy is reported throughout as Spearman ρ and relative ℓ2.
Three outcomes decide the project. First, ρ must stay within 0.01 of the retrained strongest baseline. Second, certified $L_k$ must be within a factor of about two of measured worst-case sensitivity; otherwise the bound is too loose to be informative. Third, coverage must hold after the segmenter swap without re-scoring any videos, with width tracking error better than the constant-width interval. Failing the first means Lipschitz structure costs too much accuracy. Failing the second means the certificate is vacuous. Failing the third means the compositional story adds nothing over calibrating the end score directly.
References
- Diagnosing Error in Temporal Action Detectors
- FineDiving: A Fine-grained Dataset for Procedure-aware Action Quality Assessment
- FineParser: A Fine-Grained Spatio-Temporal Action Parser for Human-Centric Action Quality Assessment
- FineCausal: A Causal-Based Framework for Interpretable Fine-Grained Action Quality Assessment
- Hierarchical NeuroSymbolic Approach for Comprehensive and Explainable Action Quality Assessment
- Metrics for Labeled Markov Systems
- Interface Simulation Distances
- Quantitative Refinement for Weighted Modal Transition Systems
- Compositional Learning of Dynamical System Models Using Port-Hamiltonian Neural Networks
- Understanding and Improving Adversarial Robustness of Neural Probabilistic Circuits
- Compositional abstraction error and a category of causal models