Surgical video models are asked to name each action as a triplet: which instrument, doing what, to which anatomy (for example "grasper, retract, gallbladder"). A few triplets dominate real surgeries, while most of the 100 possible classes appear only a handful of times. Collecting more examples of the rare ones means annotating many more hours of video, because they are rare. We propose to produce rare triplets synthetically. Instead of generating whole surgical videos from scratch, we take a real annotated clip that already shows the right anatomy being worked on, and use a video diffusion model to repaint only the region around the instrument, so that a different instrument performs a different action. A diffusion model is a generator that learns to turn random noise into realistic samples by removing noise step by step. The anatomy label is inherited from real annotation, the instrument and verb are what we ask for, and everything outside the edited region stays real footage.
This matters because the target is where current systems fail. In the CholecTriplet2022 challenge, instrument recognition reached 85.7% AP but target recognition reached only 45.9% AP. The CholecTriplet2021 challenge found the same pattern and attributes it to long-tailed tool–tissue interactions. Our claim is that anchoring generation to real clips produces tail examples whose labels can be trusted, and that this raises tail-class triplet AP on CholecT50 more than resampling, copy-paste, or unanchored video generation do. It should also partly substitute for annotated videos when few are available.
What exists and what is different
The closest work generates medical video for downstream use, but not in this form. Endora trains a latent video diffusion model with spatial and temporal transformer blocks guided by DINO attention maps, including on CholecTriplet data. It tests downstream value only as unlabeled data for PolyDiag diagnosis, and it cannot produce triplet-labelled or spatially controlled clips. Bora is a text-to-video biomedical generator with no downstream transfer experiments, and its quality degrades on multi-step procedures. SF-VD generates mask-conditioned fluoroscopy clips for guidewire segmentation, but only for a single thin structure over nearly static backgrounds, and it needs start and end masks. SegPVSG uses generative augmentation to synthesize rare relationship transitions in panoptic video scene graphs; it is the nearest conceptual neighbour. It targets natural-scene graphs, and the file gives no details on how it keeps synthetic labels correct.
Our design takes its method from a different line: insertion of synthetic content into real context in medical imaging. 3D MCGAN inserts conditioned lung nodules into real CT and uses a separate discriminator to judge how well each nodule blends with the surrounding tissue. Difftumor paints tumors into healthy CT with a mask-guided latent diffusion model and lifts SwinUNETR from 45.88% to 56.10% Dice. We carry this insertion idea into video, where the inserted content is an action rather than an object.
The same papers also limit the design. Difftumor shows that noisy synthetic mask boundaries drop U-Net Dice from 48.48% to 32.49%. We therefore never train on synthetic pixel masks; all our labels are clip-level triplets. 3D MCGAN saw a performance dip when going from 1× to 3× synthetic data, so the mixing ratio is a variable we test rather than a fixed choice.
Research Program
Generator. We fine-tune an Endora-style latent video diffusion model on CholecT50 training videos only, then turn it into a spatio-temporal inpainting model. Its inputs are a real 16-frame clip, a region to repaint (a dilated tube around the instrument across frames), and a condition made of an instrument class and a verb. Instrument locations come from a weak localizer; the ITG-Trip detector code is one source. Because the region is dilated and we produce no masks, imprecise localization does not become label noise.
Choosing edits. Each rare target triplet ⟨i, v, t⟩ is mapped to real source clips that show target t with some other, frequent instrument and verb. The edit swaps the instrument, the verb, or both. Pairs that are anatomically implausible are excluded using the triplet co-occurrence constraints of the CholecT50 label space.
Checking edits. The instrument is verified with a classifier trained on real data; the challenge results suggest instrument recognition is reliable enough for this. Verbs are harder to check. A verb classifier trained on real data will tend to reject exactly the rare verbs we want to add, so we report results both with and without verb filtering. In addition, a surgeon rates a random sample of 300 edited clips to measure how often the label is actually correct.
Downstream use. Edited clips are added only for classes in the rare-class tail. We sweep the synthetic-to-real ratio for those classes at 0.5×, 1× and 2×. The triplet model is otherwise trained exactly as the baselines are.
Baselines
The naive reference is the Rendezvous triplet model trained on real CholecT50 data alone. CholecTriplet2021 reports it at 32.7% triplet mAP. We rerun it on the official split rather than quoting the number.
The strongest existing points on this benchmark are also from CholecTriplet2021: the challenge winner at 38.1% mAP and a per-class weighted ensemble at 42.4% mAP. For CholecTriplet2022, the top recognition score is 35.0% AP and the top detection score is 4.49% AP. We apply every augmentation on two backbones, Rendezvous and the ITG-Trip model, so that any gain can be read against a strong model and not just a weak one.
We compare against three augmentation baselines on identical real data, backbone, epochs, and a tuning budget matched to ours. First, class-rebalanced resampling combined with loss reweighting, whose sampling rates are tuned to the same tail classes. Second, temporal copy-paste: instrument crops from real clips of the requested instrument and verb are pasted into the same source regions, with their motion smoothed over time in the style of DynaAugment. Third, DynaAugment itself. That paper reports up to +13.3% on UCF101 and +0.5% to +1.7% on Kinetics-400; no reported result was found on Lacuna for DynaAugment, copy-paste, or rebalancing on CholecT50.
Two ablations isolate the key ingredient, which is anchoring generation to a real clip. The first adds Endora-style whole clips generated from the triplet text, with no real source clip, at the same count and the same filtering. The second applies the same edit pipeline frame by frame with an image diffusion model, which tests whether temporal generation matters for verbs.
Beating a baseline means a gain in tail-class triplet AP that holds across three seeds, while head-class AP stays within one point of its original value. Overall triplet mAP must also improve on the ITG-Trip backbone, not only on Rendezvous.
Why not just…?
Why not just rebalance the loss or resample rare classes? Resampling replays the same few rare clips many times. That can sharpen the decision boundary, but it cannot add new visual contexts in which a target appears with a given instrument and verb. It is cheap and may capture much of the gain on classes that already have several real examples. For this reason we report results by frequency bucket. Where rebalancing matches editing, we will say so, and the claim will shrink to classes with very few examples and to unseen combinations.
Why not just copy-paste instrument crops? This is the most serious threat, because it shares our anchoring to real clips and needs no generator. It cannot render tissue deforming under traction, or the motion of cutting or coagulating tissue, and a triplet's verb is defined by that interaction. The comparison is direct: the edit regions and source clips are identical, so any gap between the two is due to generated interaction versus pasted appearance. If copy-paste ties us on verb AP, the generative component is not justified.
Why not just self-supervised pretraining on more unlabeled surgery video? Ramesh et al. show that MoCo v2 pretraining on Cholec80 lifts tool presence detection by 20.4% and phase recognition by 7.4%. They also find that gains saturate beyond roughly 40 labeled videos, and unlabeled video contains the same skewed class distribution as labelled video. We treat pretraining as complementary: our method is run on top of a MoCo v2 Cholec80 initialization, with and without editing.
Why not just generate whole clips as Endora does? Unanchored clips have no guaranteed target anatomy, and Bora's failures on multi-step procedures suggest their labels will drift. The unanchored ablation in the Baselines section settles this question.
Proposed Experiments
The main experiment is on the official CholecT50 split. We report triplet AP, component APs for instrument, verb and target, and AP by frequency bucket. The decisive check is target and triplet AP in the rarest tercile: anchored editing must beat both copy-paste and unanchored generation there.
A held-out composition protocol tests generalization to combinations never seen together. We remove all training clips of 10 triplets whose instrument, verb and target each still appear elsewhere in training. We then test whether edited clips make those triplets recognizable at all.
A label-efficiency curve connects the method to the annotation bottleneck. We train on 10, 20 and 35 real videos, with and without editing, and ask how many annotated videos the edits are worth. We expect the edits to be worth the most at the small end, before the saturation regime documented by Ramesh et al.
Finally, we report the surgeon-rated label accuracy of edited clips and the generator's FVD against Endora's reported 460.7 on Colonoscopic, and we plot downstream AP against both. This shows whether label correctness or visual realism predicts downstream benefit.
References
- CholecTriplet2022: Show me a tool and tell me the triplet - An endoscopic vision challenge for surgical action triplet detection
- CholecTriplet2021: A benchmark challenge for surgical action triplet recognition
- [inline-graphic not available: see fulltext]Endora: Video Generation Models as Endoscopy Simulators
- Bora: Biomedical Generalist Video Generation Model
- Label-Efficient Data Augmentation with Video Diffusion Models for Guidewire Segmentation in Cardiac Fluoroscopy
- SegPVSG: Panoptic Video Scene Graph Generation via Temporal Focusing and Generative Augmentation
- Synthesizing Diverse Lung Nodules Wherever Massively: 3D Multi-Conditional GAN-Based CT Image Augmentation for Object Detection
- Optimizing Synthetic Data for Enhanced Pancreatic Tumor Segmentation
- Exploring Temporally Dynamic Data Augmentation for Video Recognition
- Dissecting self-supervised learning methods for surgical computer vision