Temporal action segmentation means labeling every frame of a long video with the action being performed, such as "cut tomato" and then "place tomato in bowl." Benchmarks for this task score models against a single human's choice of the exact frame where one action ends and the next begins. Many of these transitions are gradual, so a second annotator would often put the boundary somewhere else. Part of the "error" that current models are penalized for is therefore disagreement between humans, and no model can remove it.
This proposal tests a simple hypothesis. How gradually the video's visual features change around a boundary, measured without any labels, should predict how much human annotators disagree about where that boundary is. If this holds, we can estimate the per-boundary human ceiling on benchmarks that have only one annotation. We can also stop training models to be confident about boundaries that humans themselves cannot place. The tool for measuring gradualness is a Gaussian process (a Bayesian model that places a prior over smooth functions and returns a mean and an uncertainty) fitted with a changepoint kernel. The GP is run on a small, deliberately chosen subspace of the video features, because the raw deep features are too high-dimensional and redundant for distance-based kernels to work.
Core Claim
Around each candidate boundary, we fit a GP whose covariance switches from one regime to another. We use the changepoint kernel of Offline Changepoint Detection With Gaussian Processes:
$$k(t,t') = \sigma(t)\,k_1(t,t')\,\sigma(t') + (1-\sigma(t))\,k_2(t,t')\,(1-\sigma(t')),\qquad \sigma(t) = \frac{1}{1+e^{-s(t-t_0)}}$$
In this formula, $t$ and $t'$ are frame times, $k_1$ and $k_2$ are the covariance functions before and after the change, $t_0$ is the transition location, and $s$ is the steepness. We treat $w = 1/s$, the transition width in frames, as a label-free prediction of human boundary spread.
The claim is that $w$, estimated on a task-relevant feature subspace, correlates with measured inter-annotator spread across boundaries. It should correlate better than the uncertainty signals already available from deep segmenters. We further claim that softening training targets in proportion to $w$ improves agreement with the distribution of human boundary placements, rather than merely agreement with one annotator.
What is new is the direction of the argument. SegCPGP uses the changepoint kernel to find changes in one-dimensional economic and environmental series. It says nothing about video or about humans. Here the steepness parameter is the object of interest, and it is validated against human disagreement.
Positioning Against Prior Work
The motivation comes from the agreement-ceiling literature. MRIQC argues that rater subjectivity creates a ceiling that larger models only overfit to. Should You Fine-Tune BERT for Automated Essay Scoring? finds that deep and shallow scorers stall at the same grader-reliability limit. In video, Automated Video-Based Analysis of Surgical Meta-competencies sees AUROC fall from 91.5% to 63.8% on the hardest surgical step and attributes the drop to expert disagreement. All three diagnose a ceiling after the fact. None of them predicts, for a given sample, how high the ceiling is.
The boundary-ambiguity work in segmentation also motivates the proposal but does not solve the problem. Diffusion Action Segmentation adds generic Gaussian noise to label sequences and masks boundary frames. Boundary-Aware Cascade Networks routes hard boundary frames to later stages and blocks label bleeding with barrier pooling. BaFormer predicts boundaries with a global query. All three still train against one deterministic boundary per transition.
The closest method for modeling annotators, Multi-class Multi-annotator Active Learning with Robust Gaussian Process, learns a fixed reliability for each annotator on static images. Our disagreement is different in kind: it is tied to the content of a particular moment in a video, not to the person labeling it.
The feature-redundancy work constrains the design. Gaussian Image Anomaly Detection with Greedy Eigencomponent Selection shows that in deep feature spaces the top-variance principal components encode background and domain noise. It also shows that a greedy choice of a few non-consecutive components works far better. We reuse that search, but score components by how well a changepoint GP on them recovers annotated boundaries on training videos.
Priors for Diversity in Generative Latent Variable Models motivates adding a diversity penalty so that the selected components are not redundant copies of each other. Because our selected set is small, the cubic cost of its determinant is negligible.
Research Program
The first component is data. None of the files' segmentation benchmarks provide multiple annotations, so we would re-annotate boundaries on all 50 videos of 50Salads and all 28 of GTEA. Each video would get five independent annotators, and the original class sequence would be given to them so that only boundary placement varies. This mirrors the re-labeling logic of Are pathologist-defined labels reproducible?. For each boundary, human spread is the standard deviation of placements across the six annotations (five new plus the original).
The second component is the estimator. We start from frozen I3D features and select 5–20 eigencomponents by greedy search on training folds, with the diversity term as a tie-breaker. On each boundary proposed by a base segmenter, we fit the changepoint GP in a window of a few hundred frames. The before and after regimes use Matérn kernels, following the robustness to noisy inputs reported in Modelling Representation Noise in Emotion Analysis using Gaussian Processes. Fitting only local windows keeps exact GP inference cheap and avoids the cubic cost over full videos.
The third component uses the estimate. We retrain MS-TCN++ and ASFormer-style models with boundary targets blurred by a Gaussian of width proportional to $w$. We also report a ceiling-aware boundary score that discounts error falling inside the predicted human spread.
Baselines
The naive reference point is a single constant boundary width shared by all boundaries, tuned on training folds. It predicts human spread as a constant, so its rank correlation with spread is zero by construction. For the uncertainty signals, being useful means beating it on calibration of predicted spread. For the training intervention, the fair version of this baseline is uniform Gaussian label smoothing with a tuned width.
The strongest segmentation baselines are MS-TCN++, which reports an F1 of 70.1 averaged over F1@{10,25,50} on 50Salads, and DiffAct. Diffusion Action Segmentation reports that its average metric on 50Salads rose from 86.2% to 87.4%. These are the systems whose boundary uncertainty we must beat. We derive three competing spread predictors from them: softmax entropy at the boundary, variance of boundary location across a five-seed ensemble, and variance across DiffAct's stochastic samples. All predictors use the same I3D features, the same cross-validation splits and the same hyperparameter search budget. Beating them means a higher Spearman correlation with human spread on held-out videos, with a bootstrap confidence interval that excludes the best of them.
The key ablation replaces the greedy subspace with the top-k principal components, a random subset of components, or the full 2048-dimensional features, all at matched k. This isolates whether feature selection is necessary for the GP estimate to work. A second ablation checks whether the GP's steepness adds information beyond a model-free measure of gradualness. It replaces $w$ with the plain width of the feature-change curve, defined as the distance between segment means across a sliding window.
Why not just…?
Why not just use the ensemble or entropy uncertainty that deep segmenters already give? This is the main threat. Those signals mix together the model's own ignorance and genuine visual gradualness. They also shrink as models improve, while human disagreement does not shrink. Still, they might track human spread well. The held-out correlation comparison settles this directly. If ensemble variance matches the GP estimate, we drop the GP. The contribution then reduces to the multi-annotator data and the ceiling-aware metric, and we would say so.
Why not just apply uniform label smoothing or the truncated smoothing loss? Truncated MSE smoothing in MS-TCN++ penalizes every frame equally. The cascade networks paper shows that uniform smoothing causes predictions to bleed across genuine sharp transitions. Our hypothesis says sharp and gradual boundaries need different widths. The tuned uniform-width baseline tests this directly on agreement with the human placement distribution.
Why not just train a network head to predict boundary variance from the multi-annotator labels? A supervised head can be trained here only on 78 videos. Worse, it cannot be applied to Breakfast or COIN, where no multi-annotator labels exist. We include it as a supervised upper reference. The label-free GP estimate matters only if it gets close to this head while needing no extra annotation.
Why not just use PCA before the GP? The greedy eigencomponent results show that top-variance components carry background texture. The matched-k ablation tests whether this also holds for boundary detection in video.
Proposed Experiments
The decisive first test uses five-fold cross-validation over the re-annotated 50Salads and GTEA videos. We compute the rank correlation between $w$ and human spread for each boundary, against all baselines above. If the correlation is not clearly positive, the core claim fails and the downstream work is not pursued.
The second test asks whether the predicted ceiling is accurate. We sum predicted spread into a per-video ceiling for F1@10, F1@25 and F1@50, and compare it with the F1 that each held-out annotator achieves against the others.
The third test retrains MS-TCN++ with $w$-scaled soft boundaries. We measure the expected boundary error under the empirical human placement distribution, plus standard F1@{10,25,50} against the original annotations. We expect gains on the first measure and no loss on the second.
Finally, we apply the estimator unchanged to Breakfast. We test whether the gap between DiffAct and MS-TCN++ concentrates on boundaries the estimator marks as sharp. If it does, it would be evidence that part of the recent benchmark progress lies inside the human noise floor.
References
- Offline Changepoint Detection With Gaussian Processes
- MRIQC: Advancing the automatic prediction of image quality in MRI from unseen sites
- Should You Fine-Tune BERT for Automated Essay Scoring?
- Automated Video-Based Analysis of Surgical Meta-competencies Using Computer Vision
- Diffusion Action Segmentation
- Boundary-Aware Cascade Networks for Temporal Action Segmentation
- Efficient Temporal Action Segmentation via Boundary-aware Query Voting
- Multi-class Multi-annotator Active Learning with Robust Gaussian Process for Visual Recognition
- Gaussian Image Anomaly Detection with Greedy Eigencomponent Selection
- Priors for Diversity in Generative Latent Variable Models
- Are Pathologist-Defined Labels Reproducible? Comparison of the TUPAC16 Mitotic Figure Dataset with an Alternative Set of Labels
- Modelling Representation Noise in Emotion Analysis using Gaussian Processes
- MS-TCN++: Multi-Stage Temporal Convolutional Network for Action Segmentation