Measuring How Hard Surgeons Handle Tumors, in World Coordinates

Some cancer recurrences are caused by the operation itself. When a tumor is squeezed, torn, or cut into pieces during surgery, live cancer cells escape and implant elsewhere, a process called iatrogenic tumor seeding. Surgeons know which behaviors are dangerous: rupturing the tumor's capsule, massaging the tumor before its blood supply is clamped, fragmenting it, and breaching the urinary tract. They cannot measure any of these while operating. The harm shows up months later as recurrence. This proposal builds a vision system that turns laparoscopic video into a quantitative record of how much the tumor was handled. The record covers where each instrument touched it, how far the tumor surface was pushed in, and how sharply the instruments accelerated near it. The key design choice is to express everything in a fixed, metric frame attached to the patient rather than to the moving camera.

The design choice comes from computer vision work on tracking people in world coordinates. That work found that estimates made relative to a moving camera tangle camera motion together with subject motion, so bodies appear to float, slide, and sink. Surgical video has the same problem in a harder form. The laparoscope pans and zooms constantly, so a camera-relative depth map shows the tumor surface "moving" whenever the lens moves. In a camera frame, true indentation of tissue cannot be separated from the surgeon repositioning the scope. Our claim is that a laparoscope-motion-free, metric record of tool and tumor-surface trajectories is required to compute handling measures that are plausibly linked to seeding. Instantaneous tool-to-tumor distance does not need this, but cumulative compression and world-frame tool acceleration do.

Core Claim

Instantaneous proximity between a tool tip and the tumor can already be read from a single stereo depth frame, and we do not claim otherwise. The quantities the seeding literature points to are different. Patras, Abrahamsson et al. associate violations of onco-surgical principles, such as tumor breach and rough handling, with worse survival after nephroureterectomy. Pan et al. implicate tumor massage before vascular control, and Zhang et al. implicate rupture. All of these are about deformation and force accumulated over time. Let $x_p(t)$ be the world-frame position of tumor surface patch $p$ at time $t$, and let $x_p^0$ be its rest position, estimated before contact. A natural handling exposure is

$$E_p = \int \mathbb{1}[\text{tool in contact with } p \text{ at } t]\;\lVert x_p(t) - x_p^0 \rVert \, dt,$$

which is the indentation of each patch integrated over the time a tool is touching it. We pair it with the peak world-frame acceleration of the tool tip within a set margin of the tumor. Both quantities can be computed only when $x_p(t)$ and the tool trajectory share a frame that does not move with the laparoscope. In a camera frame, scope motion appears as spurious indentation and spurious acceleration. The claim is falsifiable: if camera-frame versions of these measures predict expert-annotated violations and phantom capsule failures as well as world-frame versions do, the world-frame machinery is unnecessary.

Research Program

The system has three parts, and each borrows a mechanism from the human-tracking literature.

The first part provides a metric scale from rigid instrument geometry. SLAHMR recovers scale and camera motion from monocular video by using known human body proportions and gait as a "biometric ruler". Surgical instruments have a better ruler. The shaft and clevis of a da Vinci tool have fixed, known dimensions, whereas the cable-driven wrist does not behave reliably. We detect instrument keypoints with detectors trained on SurgPose, a dataset of about 120,000 da Vinci instrument instances with UV-paint keypoint labels and stereo 3D ground truth. We then fit only the rigid segments to their known lengths. In stereo laparoscopy this checks the calibrated baseline. In monocular laparoscopy, which is common in non-robotic surgery, it is the main source of metric scale. This follows the idea in MeTRAbs of reasoning in physical metric space so the method stays robust when the tool is partly out of frame.

The second part separates laparoscope motion from tissue motion. We estimate dense depth with self-supervised stereo networks of the kind developed in SADepth (a GAN with stereo reprojection losses) and M3Depth (an ICP 3D-consistency loss, running at 105 FPS). Both output depth relative to the camera. We register successive depth frames to a world frame anchored on tissue that is not being manipulated, such as retroperitoneal or pelvic surfaces away from the tumor. Instrument pixels and the currently contacted region are masked out of the registration. Endoscope-arm kinematics, when available, serve as a prior and not as ground truth. This is the surgical counterpart of the camera/subject separation in SLAHMR and GLAMR. GLAMR's motion infilling suggests how to carry tool trajectories through brief occlusions or exits from view, which matter because rough handling often happens at the edge of the field.

The third part builds a persistent tumor surface and computes exposure. The surgeon or a segmentation model marks the tumor once. Its surface is tracked in the world frame, rest positions $x_p^0$ are set before contact, and $E_p$ and the acceleration peaks are computed per patch. The result is a heat map on the tumor surface that shows where it was handled and how severely. The output is an interpretable post-operative or near-real-time record. It is not an autonomous warning system, and we do not claim it is one.

We deliberately leave gas-driven aerosolization out of scope. The 28% higher bladder recurrence after laparoscopic versus open nephroureterectomy reported by Tsuboi et al. motivates the oncological concern. However, no measurement in the file connects gas flow to observable geometry, and a flow model would be speculation.

Baselines

The naive reference point is raw robot forward kinematics: tool-tip positions computed from the da Vinci joint encoders and checked against a tumor surface registered once at the start. This is free and already available in robotic cases. Hwang et al. report kinematic tip errors of 4–5 mm from cable backlash, reduced to 1–2 mm with roll-dependent calibration tables. We will run kinematics both uncalibrated and calibrated with their procedure, on the same sequences. Our method beats this baseline if its tip error relative to stereo/UV ground truth is below the calibrated 1–2 mm.

For keypoints, the strongest existing method is the SurgPose benchmark. SurgPose reports DeepLabCut at 66.2±2.01 mAP (194.8 ms per frame), YOLOv8 at 63.3±1.23 mAP (16.4 ms), and ViTPose at 53.6±1.06 mAP (7.7 ms). The 3D baseline is SurgPose's own pipeline, which lifts 2D keypoints with RAFT stereo disparity in the camera frame. We retrain YOLOv8 and DeepLabCut on the same SurgPose splits with the same tuning budget. We then add our rigid-segment metric fitting on top of each detector, so any gain is attributable to the ruler and not to a different detector. On dense depth, the strongest reported SCARED numbers are SADepth's mean absolute depth error of 17.42 mm (Test Set 1) and 11.23 mm (Test Set 2), against 21.92 mm and 15.25 mm for Monodepth2, and M3Depth's RMSE of 9.274. We reuse these networks unchanged as the depth front end so that depth quality is held constant.

The baseline that matters most is the camera-frame ablation. It uses exactly our pipeline but computes $E_p$ and tool acceleration in camera coordinates, skipping the world registration. This isolates the core claim. No reported result was found on Lacuna for world-frame surgical tool trajectory accuracy or for any intraoperative handling-exposure measure, so the comparisons for those quantities are new. Our method beats the ablation if the world-frame measures predict phantom capsule failures and annotated violations significantly better on the same sequences. A second ablation removes the instrument ruler in monocular video, to test whether scale drift corrupts $E_p$. An end-to-end video classifier trained directly on violation labels, using a backbone of the same size, serves as the learned black-box competitor.

Why not just…?

Why not just use the robot's encoder kinematics? Even after calibration, the reported error is 1–2 mm according to Hwang et al., and SurgCalib notes that encoders often miss the true physical state of the instrument because cables stretch. Kinematics also say nothing about where the tissue is or how far it was pushed. That is the term $x_p(t) - x_p^0$, which needs vision, and it is unavailable in non-robotic laparoscopy. The baseline comparison measures exactly how large the gap is.

Why not just compute tool-to-tumor distance per frame from stereo depth in the camera frame? This is the real threat. For instantaneous proximity it is probably sufficient, and we concede that. It cannot tell tissue indentation apart from scope motion, so cumulative compression and tool acceleration are contaminated whenever the laparoscope moves. The camera-frame ablation settles the question. If it matches the world-frame version on capsule-failure prediction, we will report that the extra machinery is unnecessary and narrow our claim to monocular settings, where scale drift may still matter.

Why not just rely on containment bags and en bloc resection? These methods work where they apply. Teoh et al. describe en bloc bladder resection reducing exfoliation, and the multi-port containment system encloses tissue during morcellation. However, bags are used only after dissection, and the dissection phase is where massage and rupture occur. Our system complements these techniques and does not replace them.

Why not just train a video classifier to flag violations end to end? Labeled violation videos are scarce, and a classifier output gives surgeons no spatial account of what happened. We include such a classifier as a baseline. If it equals our measures with the available labels, the geometric pipeline's value reduces to interpretability, and we will say so.

Proposed Experiments

The geometry is tested first. On SurgPose we report keypoint mAP and 3D tip error, with and without the rigid-segment ruler. On SCARED and LATTE we report depth error with the existing networks. On new dVRK sequences recorded with a deliberately moving endoscope arm, we report world-frame drift of static anatomy in millimeters over minutes and world-frame tool-tip error against UV-paint stereo ground truth.

The decisive experiment uses ex vivo tissue and phantoms. We build tumor phantoms with a fragile capsule that holds dye-loaded beads, so that capsule failure and shedding can be measured directly by dye release. Volunteer surgeons then dissect these phantoms with varying care. We test whether world-frame $E_p$ and acceleration peaks predict release better than the camera-frame ablation, encoder kinematics, and the video classifier, using area under the ROC curve with phantoms held out.

Only if the phantom study succeeds do we move to a retrospective clinical video study of nephroureterectomy. Experts would annotate onco-surgical violations using the criteria of Patras, Abrahamsson et al., and we would ask whether exposure measures agree with those annotations and, as an exploratory endpoint only, whether they associate with intravesical recurrence.

The program fails cleanly if the camera-frame ablation matches the world-frame version on the phantoms, or if no handling measure predicts dye release better than chance.