Video and sensor streams have a useful property. What is happening in them changes slowly, so frames that are close in time usually share a label. Methods that learn from unlabeled video exploit this by pulling nearby frames together in feature space. This is called temporal coherence supervision. Label-efficient methods exploit the same property from the labeling side: one human label on a timestamp can be spread to its neighbors. Both break at the same place, which is the moment one activity or surgical phase ends and the next begins. In surgical video, coherence pretraining pairs frames across short phase boundaries and blurs exactly the transitions a phase recognizer needs (Temporal Coherence-Based Self-Supervised Learning for Laparoscopic Workflow Analysis). Label-spreading methods, for their part, choose which timestamps to label using generic criteria that do not look for transitions (Coherence-based Label Propagation over Time Series for Accelerated Active Learning).
We propose to close this loop when labels are very scarce, on the order of one labeled timestamp per few minutes of stream. An unlabeled-video signal estimates where boundaries probably are. A small budget of human queries is spent confirming or placing those boundaries. The confirmed boundaries then return to the coherence objective as places it must not smooth across. The claim is that a few labels are worth most when they tell the coherence prior where to stop, not when they label the interior of long, easy segments. This matters most for short, rare phases, where current methods do worst.
Core Claim
Under a fixed budget of single-timestamp labels, a system gets higher frame-level accuracy and much higher short-phase F1 when it allocates part of that budget to estimated boundaries and uses every confirmed boundary for two things. First, the boundary truncates label propagation. Second, the boundary blocks positive pairs in the coherence loss, and frames on either side become explicit negatives. A system that pretrains with coherence and then runs generic active learning with propagation does worse. The novelty is not propagation, which TCLP already does, and not coherence pretraining, which the laparoscopic study already does. It is the feedback path in which scarce labels reshape the unlabeled objective at the one place that objective is known to be wrong.
Research Program
The encoder is trained with a temporal contrastive loss. Frames within a window $w$ of each other are positives and distant frames are negatives, as in the first- and second-order contrastive pretraining of the laparoscopic study. On top of these embeddings we compute a boundary score. It comes from a random walk over frames, borrowed from Space-Time Correspondence as a Contrastive Random Walk, which trains patch embeddings so that a walk along a palindromic frame sequence returns to its start. We reuse only the inference-time quantity and apply it along the time axis of one stream. Let $A$ be the softmax-normalized similarity between each frame and the frames within $k$ steps of it. The boundary score is
$$b_t = 1 - \big[(A^{\top})^{k} A^{k}\big]_{t,\mathcal{N}(t)},$$
where $A^k$ walks $k$ steps forward and $(A^{\top})^k$ walks back, and $\mathcal{N}(t)$ is a small neighborhood of $t$. Inside a stable segment, the walk's probability mass stays put and returns. Near a transition, the mass spreads into the next segment and does not all come back. The hypothesis to test is that this multi-step score is less sensitive to single-frame jitter than the distance between adjacent features, because it averages over $k$ transitions.
In each labeling round, the query budget is split. A fraction $\rho$ goes to the local maxima of $b_t$, spread out so that queries are not packed into one region. The rest goes to a standard informativeness criterion. The fraction $\rho$ is tuned on a held-out split, not per test video.
After queries return, the system forms a list of boundaries in each stream. A boundary is confirmed whenever two queried timestamps carry different labels. The high-$b_t$ point between them is taken as its location.
Label propagation follows TCLP's plateau model, with one change: a plateau cannot extend across a confirmed boundary.
The coherence loss is then retrained with two changes. A positive pair is down-weighted by the probability that a boundary lies between its two frames, estimated from $b_t$ along the path. Frames on opposite sides of a confirmed boundary become hard negatives. A temporal classifier head, such as the GRU used in Self-Knowledge Distillation for Surgical Phase Recognition, is trained on the propagated labels. The loop then repeats.
Two findings constrain the design. The laparoscopic study shows the failure we target: frames from adjacent short phases are paired as positives. TCLP shows that thresholding the classifier's confidence frame by frame breaks segments into fragments, which is why we keep its plateau fitting and only clip it. Approximate Bayes Optimal Pseudo-Label Selection argues that confidence-based selection becomes overconfident when parameters vastly outnumber labels. It is the reason our propagation stops at human-confirmed boundaries rather than at confidence drops, although we do not implement its Bayesian criterion.
Baselines
The naive reference point is uniform timestamp sampling at the same budget. Its labels are propagated with a fixed-width window, and the classifier sits on a coherence-pretrained encoder that is never updated. It shows how much any selection or feedback buys.
The strongest existing method is TCLP (Coherence-based Label Propagation over Time Series for Accelerated Active Learning). It fits quadratic plateaus to confidence trajectories and plugs into entropy, core-set and BADGE query selection. It reports reaching 85 to 92 percent of fully supervised accuracy with 0.4 to 1.0 percent of annotations across 50Salads, GTEA, mHealth and HAPT. It also reports up to a 7.1-fold accuracy gain on mHealth over active learning without propagation. The file has no per-dataset absolute accuracy for 50Salads, GTEA or HAPT, so we will rerun TCLP ourselves with its best query strategy per dataset. It gets the same encoder initialization, the same query budgets per round, the same number of rounds and the same hyperparameter search budget as our method.
On Cholec80 the reference numbers are those of the laparoscopic study. With 20 labeled videos, temporal-coherence pretraining raises phase F1 from 67.8 to 78.6, and the full-data peak is 84.6. Self-KD reaches 85.61 F1 and 93.24 percent accuracy with full labels. These are full-video-label regimes, not timestamp budgets. We use them as ceilings and convert our budgets to "equivalent fully labeled videos" when we report results.
Beating TCLP means higher mean accuracy at matched budgets on at least three of the five datasets, with non-overlapping seed intervals. It also means a clear gain in F1 on short phases and in boundary F1 on Cholec80.
Three ablations isolate the key ingredient. The first keeps boundary-targeted queries and clipped propagation but freezes the encoder, so there is no feedback into the coherence loss. The second keeps the feedback but uses TCLP's generic queries. The third replaces the random-walk score with a smoothed distance between adjacent features.
Why not just…?
Why not just shrink the coherence window so positive pairs rarely cross boundaries? This is the most serious threat. A small enough $w$ removes most cross-boundary pairs with no labels at all. The cost is that positives become near-duplicates, which weakens the invariance that made pretraining useful: the 67.8 to 78.6 F1 gain came from pairing frames that differ. We will sweep $w$ for the unlabeled-only baseline and report its best value per dataset. If some small $w$ matches our short-phase F1 without hurting overall accuracy, the feedback path is unnecessary, and we will report it that way.
Why not just detect boundaries with change-point detection on pretrained features and query there, with no feedback? This is what the first and third ablations test together. If adjacent-feature change points select boundaries as well as the random-walk score, and the frozen-encoder variant matches the full loop, then the contribution reduces to a query heuristic. We would then present it as that. Our expectation of a gap rests on the laparoscopic finding that the pretrained features themselves blur short phases. Change points computed on blurred features should miss exactly those phases, and only retraining can repair them.
Why not just smooth predictions afterward with self-distillation? Self-KD reduces flickering between phases and matches full-data performance using 75 percent of labels. But it smooths outputs. It cannot recover a boundary the features never separated, and it assumes that most labels are available. We will apply Self-KD on top of both TCLP and our method, to check whether its gains simply add to ours or replace them.
Proposed Experiments
We run on the four TCLP benchmarks (50Salads, GTEA, mHealth, HAPT) and on Cholec80 phase recognition. For Cholec80 we pretrain on unlabeled training videos and simulate timestamp queries from the ground-truth phase annotations. Budgets span 0.2 to 2 percent of frames, over five rounds and three seeds.
The metrics are frame accuracy and macro F1, boundary F1 within a tolerance window, and per-phase F1 on the short Cholec80 phases that the laparoscopic study identified as failure cases. We also count how often positive pairs straddle true boundaries before and after feedback. This last count is the mechanism check: if it does not drop, any accuracy gain has some other cause.
The decisive checks are these. First, the full loop must beat TCLP at matched budgets. Second, it must beat the frozen-encoder ablation, which shows that feedback matters. Third, it must beat the best small-$w$ baseline on short-phase F1, which shows that the labels do something that tuning the window cannot. Fourth, on Cholec80 we will report the smallest timestamp budget at which the method matches the 78.6 F1 obtained with 20 fully labeled videos. That budget, stated in annotated frames, is the practical headline.
References
- Temporal Coherence-based Self-supervised Learning for Laparoscopic Workflow Analysis
- Coherence-based Label Propagation over Time Series for Accelerated Active Learning
- Space-Time Correspondence as a Contrastive Random Walk
- Self-Knowledge Distillation for Surgical Phase Recognition
- Approximately Bayes-Optimal Pseudo Label Selection