In the early days of surgical AI, researchers focused on "phase recognition"—identifying broad stages of an operation, such as "dissection" or "suturing." While useful for hospital logistics, this high-level view lacks the granularity needed for real-time robotic assistance or detailed skill assessment. Modern research has shifted toward a more precise framework: the action triplet.
The Anatomy of a Triplet
An action triplet decomposes a surgical moment into three fundamental components: \<Instrument, Verb, Target>. For example, instead of simply labeling a frame as "surgery," a model must identify that a grasper (instrument) is retracting (verb) the gallbladder (target).
As Chinedu Innocent Nwoye and colleagues demonstrate in the CholecTriplet2021 challenge, this shift moves AI from simple image classification to a deep understanding of surgical interactions. Some researchers, such as A. Schulze, have even proposed "quadruplets" by adding the Actor (e.g., lead surgeon vs. assistant) to better coordinate collaborative robots.
The Long-Tail Problem
The primary obstacle in training these models is the "long-tail" distribution of surgical data. In any given procedure, certain actions are ubiquitous—a grasper holding tissue occurs thousands of times. However, the most critical maneuvers—clipping a specific artery or managing a rare anatomical anomaly—may only appear for a few seconds across hundreds of hours of video.
Vivek Singh Bawa and the team behind ESAD point out that this natural imbalance causes standard models to become overconfident in frequent, routine tasks while failing to recognize the rare, high-stakes events that matter most for patient safety. This is not just a data shortage; it is a fundamental bias where the "noise" of common actions drowns out the "signal" of rare ones.
From Classification to Spatial Detection
To solve this, the field is moving beyond identifying if a triplet exists in a frame to where it is happening. Zhixi Li and Yiliang Chen introduced ProstaTD, a dataset that provides nearly 200,000 bounding box annotations. By forcing models to localize the instrument and the target simultaneously, researchers can use "instance-level self-distillation" to help the AI focus on the unique spatial relationships of rare actions, rather than just memorizing the most frequent pixels.
This spatial grounding is crucial because surgery is inherently hierarchical. As Natalia Valderrama explains in Towards Holistic Surgical Scene Understanding, a specific action triplet is always nested within a step, which is nested within a phase. Understanding the "where" helps the model infer the "what" even when the data is scarce.
Bridging the Gap with Language and Multimodality
Recent breakthroughs attempt to bypass the long-tail bottleneck by using semantic logic. Pengpeng Li and others have developed vision-language frameworks that treat surgical actions like sentences. By modeling the linguistic relationship between "grasper" and "retract," a model can "guess" a rare triplet it has seldom seen by relying on its understanding of how those components behave in other contexts.
Furthermore, work by Chantal Pellegrini on the MM-OR dataset suggests that the "tail" of the distribution becomes easier to manage when we look beyond the endoscopic camera. By fusing video with 3D point clouds and robotic logs, AI can use the physical state of the operating room to disambiguate rare maneuvers that look visually similar to common ones.
The goal of this research direction is a system that doesn't just see a video feed, but understands the structured, hierarchical, and often rare interactions that define successful surgery.