Surgical video models are asked to name what a tool is doing to which tissue, for example "grasper, retract, gallbladder." The instrument and the organ are usually visible in a single frame. The action is not: it lives in how the tool moves over a few seconds. Many actions are also rare. When a model sees only a handful of examples of a rare action, the cheapest way to lower its training loss is to memorize what the scene looks like when that action tends to happen, such as the lighting, the organ in view, or the stage of the operation. It does not learn the motion itself. This note proposes a simple fix that needs no image generator. For each rare-action clip, find a real clip from the same video, a few seconds away, that shares the same instrument, camera, organ, and lighting but carries a different action label. Then train the model to tell the two apart. Because almost everything static is held constant inside such a pair, only the motion can separate them.
The opportunity sits where four known problems meet. Fine-grained vision models tend to latch onto global keywords instead of the detail that matters. Surgical models fall into "spurious traps" on background cues. Single frames cannot encode direction or velocity. Tail classes collapse under ordinary training. Existing surgical work treats these problems one at a time. Temporal fusion has been added without any pairing objective, imbalance has been handled with soft labels on single frames, and robustness has only been diagnosed, never trained for. Same-video minimal pairs address all four with one training signal.
Core Claim
Our claim is that tail-class failure in surgical action triplet recognition is largely a temporal shortcut problem, not just a frequency problem. A contrastive objective whose negatives are temporally adjacent, same-video clips with different labels should help most on exactly the classes where static co-occurrence is the easiest shortcut: rare, motion-defined verbs. It should also make predictions more stable under visual corruption, because the features it rewards are motion features rather than scene features.
The idea borrows the logic of counterfactual pairing from vision-language work. CounterCurate builds image-caption pairs that differ along one axis and reports large gains on spatial and semantic compositional benchmarks. Its synthetic images agree with their intended captions only about 85% of the time, so roughly 15% of its pairs are noisy. Its pairs are also static images. We replace generated counterfactuals with ones that occur naturally in the video. This removes the generative-artifact problem the file documents for SurgCSS and SurgFlowVid, and it moves paired differentiation into video, which the file flags as largely unexplored.
Why the Gap Is Real
Three papers motivate the proposal. Why Deep Surgical Models Fail? shows on CholecT45 that triplet models rely on spurious background attributes. Under its perturbation protocol, the Rendezvous model with a DenseNet-121 backbone holds 26.3% average precision while the Swin-T version falls to 20.4%. That analysis is static-only and offers no fix. Self-distillation for surgical action recognition wins the CholecTriplet 2022 challenge with soft labels on single frames, and its authors name temporal context as the missing piece. Rendezvous in Time adds a causal six-frame temporal attention module. Its gains are concentrated on motion-heavy verbs ("pack" +12 AP, "aspirate" +7.1 AP), while overall triplet mAP moves only from 28.8% to 29.7%. Read together, these results suggest that temporal input helps where motion is decisive, but a classification loss alone does not force the model to use that input instead of scene shortcuts.
Two papers constrain the design. The long-tailed chest X-ray benchmark of Holste et al. finds that classifier re-training (cRT) wins among re-balancing methods, and that MixUp-style blending destroys the fine detail that rare findings depend on. We therefore do not blend clips. We keep cRT as a component and as a baseline. The Action Tubelet Detector finds that gains from stacking frames saturate around six frames, which RiT echoes with its $m=6$ window. We therefore work with short clips and do not claim long-horizon modeling.
Research Program
The backbone is a clip encoder, a network that takes a short window of frames and outputs one feature vector. We use six frames at 1 fps to match RiT, with a frame encoder initialized as in the self-distillation work and a light temporal module on top. The new component is a pair-mining procedure and a loss built on it.
Pair mining runs per video. For each anchor clip whose triplet label has been stable for at least $k$ seconds, we look within a window of $\pm\Delta$ seconds for clips whose verb differs but whose instrument matches. Matching the instrument means the pair varies the action rather than the tool. Requiring label stability on both sides removes clips straddling a boundary, where 1 fps labels are least reliable. Anchors are drawn with a class-balanced sampler, so rare triplets supply most anchors and their negatives come mostly from frequent classes in the same scene. This is the "head-tail" pairing, and it is where we expect the imbalance benefit. As a second kind of negative, we shuffle the frame order of the anchor clip. A shuffled clip has exactly the same static content but no coherent motion, so the model should not embed it like the original.
Let $z_a$, $z_p$, $z_n$, and $z_s$ be normalized clip features for the anchor, a same-label positive from another video, the same-video minimal-pair negative, and the shuffled anchor. Let $\mathrm{sim}$ be cosine similarity, $\tau$ a temperature, and $\mathcal{N}$ the set formed by $z_n$, $z_s$, and ordinary in-batch negatives. The pairing loss is
$$\mathcal{L}_{\text{pair}} = -\log \frac{\exp(\mathrm{sim}(z_a,z_p)/\tau)}{\exp(\mathrm{sim}(z_a,z_p)/\tau) + \sum_{z \in \mathcal{N}} \exp(\mathrm{sim}(z_a,z)/\tau)}.$$
It is added to the usual multi-label triplet loss. The positive comes from a different video so that the shared content of a pair cannot be scene identity. After training, the classifier head is re-trained with balanced sampling (cRT). The shuffled negative is down-weighted for verbs where frame order carries little information; per-verb results will show where it matters.
The analysis has three parts. The first is a static-reliance gap: for each class, the drop in accuracy when test clips are frame-shuffled or replaced by a single repeated frame. If the claim holds, our model should show a larger drop than the baselines, meaning it actually uses motion, and the effect should be largest on tail verbs. The second is a corruption sweep at test time with synthetic smoke, motion blur, and specular glare, together with the perturbation protocol of Cheng et al. The third is a check that the model looks at the right place, using CholecTriplet-Seg from Grounding Surgical Action Triplets: we measure the fraction of class-activation mass that falls inside the true instrument mask.
Baselines
The naive reference is a single-frame Swin-T trained with standard loss. The self-distillation paper reports 32.3% triplet mAP for this model under CholecT45 cross-validation. We rerun it ourselves on the official splits so that every comparison uses identical data.
The strongest existing single method is the self-distillation ensemble of Yamlahi et al.. It reports 38.5% triplet mAP on CholecT45 cross-validation and 37.4% on the CholecTriplet 2022 test set. It is the strongest competitor because it already targets imbalance and label ambiguity and holds the best reported numbers. We will reproduce a single-model version and add our pairing loss on top, so the comparison measures what the pairing contributes rather than what ensembling contributes. Ensembles are compared only against ensembles of the same size.
The strongest temporal competitor is Rendezvous in Time. It reports 64.0% verb mAP and 29.7% triplet association mAP, but on a different backbone from the self-distillation result. To avoid mixing protocols, we will put RiT's temporal attention module on our backbone and train it with classification loss only. We will also train a combination of that temporal backbone, self-distillation, and cRT. This combination is the fair "assemble existing parts" competitor, and it is the result to beat. All systems receive the same tuning budget, measured in hyperparameter trials and GPU-hours, on the same splits, with three seeds each.
Two ablations isolate the key ingredient. In the first, same-video adjacent negatives are replaced by random negatives of a different class from other videos, keeping the loss and the sampler unchanged. This is ordinary supervised contrastive learning with class-balanced sampling. In the second, the shuffled negative is removed. Beating the combined baseline means a gain in tail-class mAP whose 95% confidence interval excludes zero across folds and seeds, with no loss in head-class mAP, and the random-negative ablation must show most of that gain disappearing. If random negatives match same-video negatives, the core claim is wrong and we will say so.
Why not just…?
Why not just add a temporal module and re-balance the classifier? This is the most serious threat. RiT shows that temporal fusion helps motion-heavy verbs, and Holste et al. show that cRT is a strong re-balancer. Their combination may capture most of the available gain. Nothing in the file rules this out. RiT's small overall triplet gain (28.8% to 29.7%) hints that temporal input alone is underused, but that is suggestive, not decisive. The combined baseline above is built to settle the question directly. The static-reliance gap gives a second, mechanism-level test: if the combined model already relies on motion as much as ours does, our contribution is redundant.
Why not just synthesize more tail clips with a video diffusion model? SurgFlowVid reports 10–20% Jaccard gains on under-represented surgical actions, so generation clearly helps. It also has documented drift and artifacts, needs heavy pre-training, and trains the downstream model with ordinary loss. Extra tail clips can still carry the same scene shortcut. The two approaches are complementary, not exclusive. We cannot reproduce SurgFlowVid on CholecT45 within this budget, and the file gives no reported CholecT45 result for it, so we do not claim to beat it. We position minimal pairs as a generator-free alternative whose gains, if real, should add to synthetic augmentation.
Why not just train with corruption augmentation for robustness? Adding smoke and blur at training time teaches invariance to those specific corruptions. It does nothing about background co-occurrence shortcuts, which Cheng et al. identify as the main failure. We include corruption augmentation as an additional baseline arm and test on corruption types it was not trained on. We expect it to win on the corruptions it saw and not on the spurious-attribute perturbations; if it wins on both, the robustness part of our claim fails.
Proposed Experiments and Decisive Checks
The main experiment uses the official CholecT45 cross-validation folds and reports triplet, verb, and instrument mAP, broken down by head, medium, and tail triplet groups defined by training frequency. One final configuration, chosen on validation folds only, is evaluated on the CholecT50 challenge test split, so results can be read against the 37.4% self-distillation and 32.7% Rendezvous reference points. Robustness is measured as the relative mAP retained under each corruption at three severities, and under the Cheng et al. perturbation protocol, where 26.3% and 20.4% are the reported reference points. Grounding is measured on CholecTriplet-Seg as described above.
The decisive check is a joint condition. The pairing model must beat the combined temporal, self-distillation, and cRT baseline on tail-class mAP. The random-negative ablation must lose most of that margin. And the static-reliance gap must grow most on tail verbs. If all three hold, the gain comes from removing temporal shortcuts. If tail mAP improves but random negatives do just as well, the effect is generic contrastive regularization. If neither improves, then temporal shortcuts are not what limits rare surgical actions, and imbalance should be treated as a pure data-scarcity problem.
References
- CounterCurate: Enhancing Physical and Semantic Visio-Linguistic Compositional Reasoning via Counterfactual Examples
- Rethinking data imbalance in class incremental surgical instrument segmentation
- Mitigating Surgical Data Imbalance with Dual-Prediction Video Diffusion Model
- Why Deep Surgical Models Fail?: Revisiting Surgical Action Triplet Recognition Through the Lens of Robustness
- Self-distillation for Surgical Action Recognition
- Rendezvous in time: an attention-based temporal fusion approach for surgical triplet recognition
- Long-Tailed Classification of Thorax Diseases on Chest X-Ray: A New Benchmark Study
- Action Tubelet Detector for Spatio-Temporal Action Localization
- Grounding surgical action triplets with instrument instance segmentation: a dataset and target-aware fusion approach