Surgical workflow models label each moment of an operation with its phase, such as "dissection" or "clipping." They lose accuracy when moved to a new hospital, where the camera, lighting, instruments and surgeons all differ. The two usual remedies both fit clinical deployment poorly. The first is to densely label the new hospital's videos frame by frame and retrain, which is expensive. The second is to change the model's weights, which a certified medical device generally cannot do without recertification (Regulating AI Adaptation). This note proposes a third route. A clinician at the new hospital watches a handful of local videos and clicks once inside each phase, a "timestamp." Those clicks are spread offline into dense frame labels, and the dense labels train a small input-side adapter. The adapter rewrites incoming frames so that the frozen, causal model, which sees only past frames, performs well again.
The claim is that sparse timestamps on target-hospital video carry far more adaptation signal than the same number of labeled frames used as isolated images. The extra signal comes from what the clicks reveal: the order of phases, their approximate position in the video, and continuity anchors that can be propagated through neighboring frames. Propagating them offline sidesteps the main weakness of timestamp methods, which is that the best ones must look at future frames and so cannot run live. Here the look-ahead method is used only at adaptation time on recorded videos. The model that runs in the operating room stays causal and keeps its frozen weights.
Where this sits
The proposal combines three pieces that have so far appeared only separately. Each is listed with the role it plays here.
The first supplies the method for making labels from clicks. UCATD shows that one timestamp per surgical step, about 0.1% of frames, can be spread by uncertainty-filtered diffusion over frozen self-supervised features. It reaches 82.7% accuracy on open orthopaedic surgery, against 80–88% for fully supervised models. Its diffusion looks in both time directions, its causal variants lose substantial accuracy, and it was tested on a single hospital. The authors name multi-center validation as open. Order-preserving optimal transport is optimal transport, a method for assigning frames to labels at minimum total cost, constrained to respect segment order. It shows that one timestamp per segment yields dense pseudo-labels within 1–2% of full supervision on sensor data. We reuse both for label propagation, never at deployment.
The second supplies the method for adapting without touching the model. INPUTADAPTER trains an encoder-decoder that reshapes target inputs into the source distribution while the predictor stays frozen. On eICU it ties or beats end-to-end fine-tuning, with +14.1 AUCPR on acute kidney injury. It requires dense retrospective labels from the target site and was tested on health-record time series, not video. We replace its dense labels with propagated timestamps and move it to surgical video.
The third motivates the comparison. SPA adapts a frozen vision-language model to new hospitals from 8–32 labeled images per phase, plus a hand-written procedural flowchart used to train a temporal smoother. It reports 43.5% F1 on BernBypass with 32 labeled images. SPA treats its labels as static images and takes its temporal structure from a manually built graph. We take the temporal structure from the target videos themselves, through where and in what order the clicks fall. TeCNOadap and the HeiChole benchmark establish that cross-hospital degradation persists even with dense multi-center training. Their figures are 82% accuracy on an unseen hospital and a best phase F1 of 65.4% across three centers.
Two further results constrain the design. Lee and Byun show that point labels pull models toward the most salient frames and fragment the predicted segments. That failure would be worse if the adapter were trained on the clicked frames alone, which is why we propagate before training. Label Dropout shows that when sites annotate differently, networks learn the site's appearance as a cue for the label. Raw adaptive cross-entropy gave left-ventricular-myocardium Dice of only 0.319, rising to 0.518 with dropout. Its temporal analogue here is a site-specific clicking convention, for example always clicking near a phase's start, and we test for it directly.
Research Program
The setting has three parts. A source phase recognizer is trained densely at one or more hospitals and then frozen; we use a causal temporal model in the TeCNO style and also LoViT run causally. A target hospital supplies $k$ unlabeled training videos, each with one click per phase instance. Separate held-out target videos are used for testing.
Adaptation has two stages. In the first, labels are propagated offline. Frozen self-supervised frame features in the DINO style are computed for each target video; they are deliberately not the source model's features, which are the ones distorted by the domain shift. The clicks fix the ordered list of phases present in that video and one frame inside each. Order-preserving transport assigns every frame to one of these ordered segments, and UCATD-style Monte Carlo dropout removes uncertain frames near boundaries. Monte Carlo dropout means estimating uncertainty from the spread of predictions across repeated runs with random units switched off. The result is dense, confidence-weighted pseudo-labels.
In the second stage, a small residual image-to-image network $A$ is placed in front of the frozen recognizer. Its change to each frame is bounded, so it cannot invent content. It is trained by backpropagating confidence-weighted cross-entropy against the pseudo-labels through the frozen causal model, and the clicked frames themselves receive full weight. Because the frozen model is causal and $A$ processes each frame independently, the deployed system stays causal. We would also report a per-video photometric-only variant of $A$ as a minimal, easily audited adapter.
Two analyses accompany the method. The first is a click-convention stress test. Synthetic timestamps are drawn from dense labels in three ways: uniformly within each phase, biased toward phase onsets, and biased toward the most motion-heavy frames. We then measure how far each bias shifts the predicted boundaries, and whether randomizing the anchor's assumed position during propagation removes the shift, as a temporal counterpart of Label Dropout. The second is calibration: whether the adapted system's confidence matches its accuracy, given that out-of-domain overconfidence (Improved Post-hoc Probability Calibration for Out-of-Domain MRI Segmentation) is a documented hazard.
Baselines
The naive reference is the frozen source model applied to the target hospital unchanged. This is the deployment default, and every gain is measured against it. A second label-free reference is unsupervised test-time adaptation of the input adapter with an entropy-minimization objective, run under the same adapter architecture and step budget. It isolates how much the clicks contribute beyond adaptation itself.
The strongest existing competitor is SPA-style few-shot adaptation (SPA). We give it exactly the frames our annotator clicked, as labeled still images, which equalizes the click budget. We run it with its frozen vision-language linear probe and a task-graph smoother written for each procedure. It has the same target videos available for its test-time adaptation, and its hyperparameters get a search budget equal to ours. Its reported numbers are 43.5% F1 on BernBypass with 32 labeled images, and a 17.33% F1 gain on Cholec80 over earlier few-shot methods. Beating it means higher phase-level macro F1 at equal clicks on held-out target videos, with non-overlapping confidence intervals across at least three seeds and video splits. We would also note whether it holds without a hand-built graph.
The key ablation trains the identical adapter on the clicked frames only, with no propagation. Its gap to the full method measures the value of using clicks as temporal anchors rather than as sparse labels. The other key comparison is a from-scratch UCATD model trained only on the target timestamps, with no source model. It tests whether the locked source model contributes anything at all. Two upper references bound the method from above: an adapter trained on dense target labels, which is INPUTADAPTER's original regime, and unlocked end-to-end fine-tuning of the source model on dense target labels. All methods share the same frozen feature extractors where applicable, the same target video splits, and matched compute and tuning budgets.
Why not just…?
Why not just label the same frames as images, as SPA does? At an equal number of labeled frames, images carry no information about order, position or continuity. Frame-only predictions flicker for that reason, which is why SPA needs a hand-built task graph and why TeCNOadap stacks frames. This is the central experiment. If SPA matches our method at equal clicks, then embedding the labels in video adds nothing, and the proposal fails on its main claim.
Why not just train a timestamp-supervised model from scratch at the new hospital? This is a real threat. UCATD alone reaches 82.7% on its home hospital with timestamps only. If enough target videos are available, the locked source model may be dead weight. We therefore sweep the number of clicked target videos $k$ from 2 to the full training split and look for a crossover. Our expected outcome is that adaptation wins clearly at small $k$, while from-scratch training is acausal unless it accepts UCATD's reported causal penalty. If from-scratch training wins at every $k$ even under a causal constraint, the honest conclusion is that hospitals should collect clicks and retrain. In that case the contribution shrinks to the regulatory constraint alone.
Why not just fine-tune the whole model on the propagated labels? Fine-tuning changes certified weights (Wu et al.). On health records, input-space adaptation tied or beat it (INPUTADAPTER). We report fine-tuning on the same pseudo-labels as a reference. A large gap in its favor would mean that video shift is not correctable at the input, which is itself worth knowing.
Why not just train on dense multi-center data upfront? Dense multi-center training is already known to leave a gap: 82% on an unseen hospital (TeCNOadap) and a best phase F1of 65.4% on HeiChole (HeiChole). Our source models include a Cholec80-plus-HeiChole model, so we can measure whether click adaptation still helps on top of a multi-center source.
Proposed Experiments
We use two transfer pairs. The first is StrasBypass to BernBypass: the same gastric bypass procedure at a different hospital, where SPA's 43.5% F1 provides an external anchor. The second is Cholec80 to the HeiChole centers, run leave-one-center-out, with a Cholec80-plus-two-centers source for the multi-center variant. Timestamps are simulated from the existing dense labels under the three placement schemes described above. All evaluation is causal, at frame rate, on held-out target videos. We report frame accuracy, phase-level macro F1, Jaccard, a segmental edit score to capture flicker, and expected calibration error. The MHB hospital and MIPO data have no public availability stated in the file, so we do not rely on them.
Three checks decide the outcome. The first is whether the propagated-anchor adapter beats equal-budget SPA and the clicked-frames-only ablation. The second is whether it closes most of the gap between the frozen source model and the dense-label adapter. A useful target is recovering at least two-thirds of that gap with clicks on about five target videos. The third is whether biased click placement measurably shifts the predicted boundaries, and whether randomizing the assumed anchor position fixes the shift. For our exact protocol, no reported result was found on Lacuna. No reported result was found on Lacuna for timestamp-supervised adaptation on BernBypass or HeiChole, so every number in these comparisons will be measured, not quoted.
References
- Regulating AI Adaptation: An Analysis of AI Medical Device Updates
- Introducing surgical workflow recognition in orthopaedic surgery with timestamp supervision
- Timestamp-Supervised Wearable-Based Activity Segmentation and Recognition With Contrastive Learning and Order-Preserving Optimal Transport
- Adapt the Data, Not the Model: Input-Space Adaptation for Frozen Time-Series Predictors
- Recognizing Surgical Phases Anywhere: Few-Shot Test-Time Adaptation and Task-Graph Guided Refinement
- Surgical Phase Recognition for different hospitals
- Comparative Validation of Machine Learning Algorithms for Surgical Workflow and Skill Analysis with the HeiChole Benchmark
- Learning Action Completeness from Points for Weakly-supervised Temporal Action Localization
- Label Dropout: Improved Deep Learning Echocardiography Segmentation Using Multiple Datasets with Domain Shift and Partial Labelling
- LoViT: Long Video Transformer for Surgical Phase Recognition
- Improved Post-hoc Probability Calibration for Out-of-Domain MRI Segmentation