Grading Surgical Video Search the Way a Surgeon Would

Systems that search long video archives are scored almost entirely by temporal overlap: did the returned clip start and end near the annotated clip? In surgery that score is blind to what matters. Suppose a surgeon asks for "the moment the clipper closes on the cystic duct." A system that returns a well-aligned clip of the grasper merely retracting the duct scores about as well as one that returns the real clip. A system that invents a clip for a complication that never happened is not penalized at all under the usual recall metric. Text generation faced the same problem earlier. Radiology report metrics built on word overlap rewarded fluent reports that described findings the patient did not have, until structured, clinically grounded metrics replaced them. We propose to bring that lesson to video search. We will build a corpus-level moment retrieval benchmark from annotated surgical video, and a score that credits a retrieved clip according to which parts of the surgical action it got right, when it happened, and whether the action happened at all. We will fit that score to surgeon judgments and then test whether it changes which retrieval systems look best.

The contribution is a measurement result, and it can fail. If plain temporal overlap ranks systems the same way surgeons do, the structured score is unnecessary, and we will report that. If it does not, every published comparison of video indexing methods applied to interventional video is ranking systems on the wrong quantity.

Core Claim

Our claim is that, for retrieving moments of structured actions in surgical video, a score combining component-level action correctness, temporal overlap and explicit penalties for returning something when the target is absent will agree with surgeon judgments significantly better than temporal-overlap recall or frame-level average precision. We also claim the two kinds of score will disagree often enough to reorder current late-fusion indexing systems. By late fusion we mean systems that encode video offline into stored vectors and match a query against them at search time.

What Exists and What Is Missing

Two evaluation traditions sit on either side of this question without touching. The surgical side is defined by the CholecT50 action-triplet work. CholecTriplet2021 and Data Splits and Metrics for Method Benchmarking on Surgical Action Triplet Datasets describe each surgical action as an ⟨instrument, verb, target⟩ triplet, standardize the data splits, and release ivtmetrics, which computes per-component and per-triplet average precision. This work supplies our ontology and labels. It also motivates the proposal, because its metrics are strictly per-frame at 1 frame per second, operate within single videos, and have no notion of a moment with a start and an end.

The retrieval side supplies the systems we will evaluate, and it also shows the gap. TVR introduced XML, a late-fusion model that detects start and end edges over a query-to-video similarity curve. Event-aware Video Corpus Moment Retrieval indexes clustered "events" instead of frames, which cuts latency and storage. Generalized Video Moment Retrieval adds queries whose target is absent and rejects them with a fixed evidence threshold. All three score themselves with temporal overlap alone, and all three were built on open-domain video.

VERIFIED is the closest prior work. It shows that coarse queries make corpus retrieval benchmarks ambiguous, and that models drop sharply when the queries are made fine-grained. However, VERIFIED makes the queries richer and keeps the metric unchanged. We keep the queries structured and change the metric.

The methodological template comes from text. Evaluating Progress in Automatic Chest X-Ray Radiology Report Generation parsed reports into clinical entity graphs and fitted a composite score (RadCliQ) to the number of errors radiologists found. That score reached a Kendall-tau of 0.522 with radiologists and revealed that generative models were only 5.5% better than random retrieval. ASSESS likewise replaced string matching with a structured edit distance for formal mathematics and validated it against experts. We reuse their protocol: define an error taxonomy, collect expert ratings, fit a small composite, and test it against simpler scores. The difference is that we apply it to temporal retrieval, where a prediction is a span in time rather than a string.

Research Program

The benchmark is built from CholecT45, the public subset of CholecT50. It has 45 cholecystectomy videos, 100 triplet classes and 1 fps labels. We convert frame labels into moments by merging consecutive frames that carry the same triplet, bridging gaps of up to two seconds. Queries come in three kinds. Triplet queries are rendered as templated and paraphrased language. Procedural queries tie one triplet to an earlier one, for example clipping after dissection of the same target. No-target queries name a triplet that is absent from a given video, or absent from the whole held-out fold.

The corpus is roughly the 151,000 annotated frames, which is only about forty hours of video. That is small by indexing standards, so scale claims will be limited to this size. We will report latency-versus-corpus-size curves by adding unlabeled distractor video, if such video can be obtained. Splits follow the published 5-fold cross-validation protocol.

Before fitting the score, we collect human ratings. For a pool of retrieved moments sampled from all baseline systems, several surgeons each label the error type. The categories adapt the radiology taxonomy: wrong instrument, wrong verb, wrong target, boundary too early or too late, missed moment, and moment returned for an absent event. Each surgeon also gives a 1–5 usefulness rating. We then fit a clinical moment score to these ratings. For a retrieved span $\hat{s}$ with predicted triplet evidence against ground-truth span $s$:

$$ \mathrm{CMS}(\hat{s}, s) = \mathrm{tIoU}(\hat{s}, s)^{\alpha} \cdot \Big(1 - \sum_{c \in \{i,v,t\}} w_c\, e_c\Big) - \lambda\, \mathbb{1}[\text{target absent}] $$

Here $\mathrm{tIoU}$ is temporal intersection-over-union, and $e_c \in \{0,1\}$ marks an error in instrument, verb or target, judged by the visual content of the returned span. We determine $e_c$ from the dense frame labels inside $\hat{s}$, so no model judges another model. The nonnegative weights $w_c$, the exponent $\alpha$ and the false-positive penalty $\lambda$ are fitted by ordinal regression to surgeon ratings on a calibration fold. Corpus-level scores average the best-ranked CMS over the top $K$ results. The functional form is deliberately small, with five parameters. That keeps it auditable and makes it hard to overfit a few thousand ratings.

Baselines

The retrieval systems we evaluate span the space the file documents. They are run on the same benchmark, receive the same features where their design permits, and get the same hyperparameter-search budget on the calibration fold.

The naive reference point is CLIP-style zero-shot frame retrieval: cosine similarity between the query text and frame embeddings, with a sliding window producing spans. It shows what generic vision-language features do with no surgical knowledge.

The strongest domain-specific competitor is a frame classifier turned into a retriever. We run Rendezvous, the triplet recognizer from Data Splits and Metrics, which reports 29.4% triplet mAP on the 5-fold split, and apply temporal smoothing and thresholding to extract spans. This baseline is dangerous because it already carries the surgical ontology, and only temporal post-processing is added.

The strongest general indexing methods are XML from TVR and the event-level index from Event-aware Video Corpus Moment Retrieval. On TVR they report 7.25% Recall@1 at IoU 0.5 and 10.12 Recall@1 at IoU 0.7 respectively. Both are retrained on the surgical corpus. For no-target rejection we run BCANet from Generalized Video Moment Retrieval, which reports 77.92% no-target accuracy on NExT-VMR, with its threshold tuned on the calibration fold. A long-context multimodal model prompted for timestamps rounds out the set. MomentSeeker reports 29.6 R@1 for Gemini 2.5 Pro on long open-domain video, and this baseline tests whether such models do better or worse on specialized content. We found no reported result on Lacuna for any of these systems on surgical moment retrieval.

The metrics being compared are the real object of study. The first is standard Recall@K at IoU thresholds of 0.3, 0.5 and 0.7. The second is frame-level ivtmetrics triplet AP computed inside the returned spans. The third is Recall@K reported alongside a separate no-target accuracy, as in the generalized retrieval work. Ablations of CMS isolate its ingredients: uniform component weights instead of fitted ones, the temporal term alone, the component terms alone, and $\lambda = 0$. "Beating" a baseline metric means a Kendall-tau with pooled surgeon ratings that is higher by a margin whose 95% paired-bootstrap interval excludes zero. The 0.522 from radiology is a sanity anchor, not a comparable target.

Why not just…?

Why not just report standard Recall@IoU on a surgical version of the benchmark? This is the real threat, and the experiment is designed around it. If component errors and temporal misalignment are highly correlated in practice, because models that pick the wrong verb also return the wrong span, then tIoU already captures most of what surgeons care about. The radiology result motivates the doubt, since overlap scores diverged from experts there, but that result was for text, not time. We settle the question by comparing correlations with surgeon ratings, and by counting how often the two metrics disagree on pairwise system rankings. If neither test separates them, we will recommend plain tIoU plus a no-target rate.

Why not just aggregate frame-level ivtmetrics over the returned span? This metric already knows the triplet ontology. Its weakness is structural. It counts frames, so a flickering prediction scattered across a moment earns partial credit that a surgeon looking for that moment would not give, and it has no term for a span returned when nothing happened. The ablation that keeps component terms but drops the temporal exponent tests exactly this. If span-averaged AP matches CMS agreement, the extra temporal structure is not earning its place.

Why not just ask a multimodal language model to judge each retrieved clip? Model judges are cheap. However, the layperson-evaluation findings cited in the file suggest fluent judges miss omissions and clinically hazardous errors, and a video model judging surgical clips inherits the weak verb and target recognition documented on CholecT50 (about 60% and 40% AP). We include a model judge as an additional metric baseline, scored against surgeons like the others. We do not assume it loses. If it matches surgeon agreement, it becomes a scalable substitute for the hand-fitted score, and that is a useful finding in itself.

Proposed Experiments and Decisive Checks

The first decisive check is agreement. Across all rated retrieved moments, CMS fitted on calibration folds must correlate with held-out surgeon usefulness ratings more strongly than Recall@IoU, span-averaged triplet AP, and the model judge, with bootstrap intervals separating it from each. Inter-surgeon agreement sets the ceiling.

The second check is consequence. We rank all baseline retrieval systems under each metric and report pairwise ranking inversions. The central prediction is that the frame-classifier retriever and the general late-fusion indexers swap places. The classifier should look worse on tIoU because of fragmented spans, and better on CMS because of correct triplets. Separately, systems without rejection should drop under CMS on no-target queries.

The third check is robustness. We refit weights using each surgeon alone and each fold alone, and report how stable $w_c$ and $\lambda$ are. A metric whose weights change sign between surgeons is not a metric.

Finally, we report query latency and index size for each system on the corpus, so that metric gains are never read apart from the cost that makes indexing worthwhile.