Trocar-Anchored Monocular Tracking of Occluded Surgical Instruments

A laparoscopic instrument is a rigid rod that enters the body through a small incision, so every motion it makes pivots about a nearly fixed point at the body wall. This proposal treats that pivot point, the remote center of motion (RCM), as a 3D landmark the tracker estimates and keeps over time. A single camera cannot tell how far away things are. Once the pivot is known, though, most of that depth ambiguity disappears. The instrument's 3D pose is then a rotation and an insertion depth about a known point, and the visible evidence only has to pin down what remains. That remainder is small enough to estimate through smoke, blood, a crossing tool, or a tip that is a few pixels wide.

The opportunity matters because today's instrument pose estimators work frame by frame and fail in the moments that matter most clinically. When the shaft is hidden or the tip shrinks to a handful of pixels, per-frame keypoint detectors and solvers produce no answer or a badly wrong one. These are the same moments in which segmentation models are known to collapse under low light and smoke. A persistent geometric anchor helps in a way appearance-based fixes do not. It comes from geometry rather than from texture, so it should carry across hospitals, scopes, and corruptions.

Core Claim

Monocular 3D instrument pose is ill-posed per frame: six degrees of freedom must be recovered from a 2D image. Suppose a tracker estimates the RCM point $r$ from the history of frames in which the shaft was clearly visible. Then each new frame is constrained much more tightly. The 2D image of the shaft fixes which viewing plane (through the camera center and $r$) contains the instrument. The 2D tip position fixes one more constraint. Only one scalar per frame stays ambiguous: the tilt of the tool within that viewing plane, which is equivalent to the tip's depth.

That one scalar can be read from any of several weak metric cues. These include the apparent shaft diameter, the foreshortened wrist-to-tip length (both have known physical size), and temporal continuity. In contrast, per-frame methods need both shaft edges and clean keypoints simultaneously.

The claim is testable. With the anchor, 3D tip error should stay roughly flat as occlusion length grows. Without it, a temporal model drifts once the occlusion outlasts its context window. The anchored model should also degrade less under physical corruptions, because each corruption removes evidence for one scalar rather than six.

Each frame $t$ in which the shaft is well observed yields a noisy 3D axis estimate with a point $p_t$, a unit direction $d_t$, and a confidence $w_t$. These come from cylinder-edge inversion with the known shaft radius, as in prior per-frame work. The anchor is the confidence-weighted point closest to all these axes:

$$ r^{*} = \arg\min_{r} \sum_{t} w_t \left\| (I - d_t d_t^{\top})(r - p_t) \right\|^2 , $$

where $I$ is the 3×3 identity, so $(I - d_t d_t^{\top})(r - p_t)$ is the perpendicular offset of $r$ from axis $t$. Per-frame depth errors along the camera axis are large but differ in direction across frames as the tool pivots. Pooling many of them should shrink the anchor's error roughly as the inverse square root of the number of well-conditioned frames. That rate follows from averaging approximately independent per-frame errors. It breaks down in a known case: pure insertion without pivoting gives parallel axes and an unobservable $r$. The tracker must detect this degeneracy and report it.

Relation to Prior Work

The anchored parameterization is borrowed, not invented. Differentiable Rendering-based Pose Estimation for Surgical Robotic Instruments searches for pose in a 4D "LookAt" space around the RCM, using differentiable rendering against SAM2 masks. That paper shows the RCM constraint keeps optimization out of local minima. It also constrains our design. It treats each frame as a slow one-shot optimization, depends on clean masks, and drifts during continuous motion. We instead estimate the anchor online from accumulated shaft evidence and use it as persistent state that carries pose through occlusion, with a feed-forward per-frame model rather than rendering.

The per-frame evidence module reuses Efficient Surgical Robotic Instrument Pose Reconstruction in Real World Conditions Using Unified Feature Detection. That method combines DINOv2 keypoint heatmaps, a Hough-space shaft-edge head, and a cylinder-inversion solver, and reaches about 130 ms per frame. It needs continuous shaft-edge visibility and was evaluated on 290 static frames. Our system turns its outputs into confidence-weighted measurements of a temporally persistent state.

On the temporal side, 3D Human Pose Estimation with Spatial and Temporal Transformers lifts 2D keypoint sequences to 3D with attention over an 81-frame window. MissFormer adds explicit missing-observation flags so attention ignores absent measurements. We reuse both as the temporal machinery.

These papers motivate the gap from opposite sides. Temporal lifters drift toward mean poses once occlusion outlasts the window. Glimpse argues that single-frame structural sampling along limbs suffices, reporting 39.3 mm MPJPE on Human3.6M. Our view is that a temporally estimated geometric anchor gives a long-memory constraint that neither a finite attention window nor a single frame provides.

Two lines of evidence motivate the corruption and domain-shift regime. SegSTRONG-C reports UNet falling from 0.94 to 0.57 DSC, and SAM2 to 0.43 DSC, under low brightness. Towards markerless surgical tool and hand pose estimation reports roughly 66 mm drill-tip error from frame-independent estimation. SurgPose supplies data but evaluates only frame-wise detectors and resolves depth with stereo disparity.

Endo3R is complementary. It recovers endoscope ego-motion from monocular video while discarding instruments as outliers. We use its camera trajectory to move the anchor into each new camera frame when the scope moves.

Research Program

The system has three parts.

The first part is a per-frame evidence model. A DINOv2 backbone with low-rank adapters follows Surgical-DINO, which updates 0.17% of parameters. It emits tip, wrist and jaw keypoint heatmaps, shaft-line parameters, and apparent shaft width. Each output carries a calibrated confidence, and each output is flagged as missing in the MissFormer style when confidence is low.

The second part addresses lost tiny features, since the tip and jaw are what downsampling erases. Rather than a general learned pooling operator, the current tracked 3D state predicts where the tip should appear. A second pass then reads a native-resolution crop at that location. Geometry tells the network where full resolution is worth spending. LIP: Local Importance-Based Pooling raised small-object AP on COCO from 21.9% to 24.0% by learning where to keep detail. It is the natural content-only alternative and is tested head to head.

The third part is the anchored state estimator. The anchor $r^{*}$ is solved in closed form over a long buffer of confident frames and refreshed slowly. For moving-scope segments it is transformed by the endoscope ego-motion, taken from robot kinematics on the dVRK or from Endo3R otherwise. A small temporal transformer over flagged measurement tokens then predicts the per-frame state given $r^{*}$: two pivot angles, insertion depth, roll, and jaw opening. Depths are expressed in focal-length-normalized units, following the virtual-depth idea of MAGneT-3D, so that scopes with different intrinsics are not confused with different geometry.

Training uses SurgPose's left images only, with 3D labels from its kinematics and stereo. Synthetic occlusion masks are applied with controlled durations.

Baselines

The naive reference point is the per-frame unified-feature pipeline of Efficient Surgical Robotic Instrument Pose Reconstruction in Real World Conditions Using Unified Feature Detection, holding the last valid pose when it fails. This shows what doing nothing temporal costs.

The first strong competitor adds a constant-velocity Kalman filter over its 6-DoF output. This is the obvious engineering fix and isolates whether learned temporal reasoning matters at all.

The second strong competitor is a PoseFormer-style temporal lifter with MissFormer missing tokens (3D Human Pose Estimation with Spatial and Temporal Transformers, MissFormer). It is trained on the same SurgPose keypoints and outputs unconstrained 3D keypoints. It is the most direct threat, because it could in principle learn the pivot implicitly within its window. It is also the exact ablation of our key ingredient: same evidence model, same transformer size and training budget, no anchor.

An accuracy reference with no real-time claim is the offline RCM-anchored differentiable renderer of Differentiable Rendering-based Pose Estimation for Surgical Robotic Instruments. It is run on a subsample of frames.

Two upper bounds frame the gap. The first is SurgPose's stereo triangulation; the second is our tracker given the true RCM from robot kinematics.

All learned baselines share the backbone, adapter rank, training data, augmentations, and an equal hyperparameter search budget on a held-out SurgPose trajectory split.

For reference, the strongest reported SurgPose numbers are 2D keypoint MAP of 66.2 ± 2.01 for DeepLabCut and 63.3 ± 1.23 for YOLOv8 (SurgPose). Liang et al. report PnP translational errors of 50 to 60 mm on dVRK frames. No reported result was found on Lacuna for monocular 3D tip error on SurgPose, so all 3D baselines will be run by us.

Beating the baselines means three things. Clean-frame error must be matched. 3D tip error must be lower, with paired bootstrap confidence intervals excluding zero, for occlusions longer than the lifter's window. Error growth with occlusion length must be visibly flatter.

Why not just…?

Why not just read the robot kinematics? On a research dVRK, forward kinematics plus a calibration gives the pose directly, and we use that as an upper bound. Clinical da Vinci video rarely comes with accessible kinematics, and manual laparoscopy has none. Monocular video is the only universally available signal. The kinematic upper bound tells us how much of the gap the anchor closes.

Why not just use stereo? SurgPose resolves depth with stereo disparity, and on stereo systems with clean images that is likely better. Most laparoscopes are monocular, and stereo matching relies on the same texture that smoke and low light destroy. We report stereo as an upper bound on clean SurgPose frames. We also report its degradation on the corrupted SegSTRONG-C sequences, which were recorded with the same dVRK stereo setup. If stereo stays accurate there, the method's value is confined to monocular scopes, and we will say so.

Why not just Kalman-filter a good per-frame estimator? A constant-velocity filter extrapolates motion. It does not know that the tool must pivot about a fixed point, so during long occlusions it drifts along exactly the depth direction that monocular estimates get wrong. MissFormer's comparison against LSTMs (0.53 m vs 0.70 m ADE on BIWI/UCY) suggests naive recurrence is beatable. The decisive evidence is the occlusion-length sweep against our filtered baseline.

Why not just a longer temporal transformer? This is the real threat. A long enough window might learn the pivot implicitly. Our answer is empirical. We give the lifter the longest window that fits the latency budget and test occlusions beyond it. We also test transfer to unseen port placements, where an implicitly memorized pivot should fail and an estimated one should not.

Why not just fine-tune or augment for domain shift? SegSTRONG-C shows augmentation-trained UNets still collapse to 0.57 DSC under low brightness. Our claim is not that the visual model becomes robust. Our claim is that the geometric state needs less from it. Augmentation is applied equally to all methods, so it cannot explain any gap.

Proposed Experiments

Experiment 1 is an occlusion-length sweep on held-out SurgPose trajectories. Masks hide the shaft, tip, or both for durations from a few frames to several times the lifter's window. We report 3D tip error in millimeters, shaft-axis angular error, and jaw-angle error. Errors are also split into depth and in-plane components, to show the anchor removes depth error specifically.

Experiment 2 measures anchor convergence. It records RCM error against the number and angular spread of confident frames, checks the predicted inverse-square-root trend, and quantifies the pure-insertion degeneracy and how reliably the system flags it.

Experiment 3 tests physical corruption on SegSTRONG-C, which provides smoke, bleeding, and low-brightness replays of the same robot trajectories. Wherever logged kinematics are available, they serve as reference 3D labels. Otherwise the clean replay's estimate serves as a pseudo-reference, so error can be attributed to corruption rather than motion.

Experiment 4 tests in-vivo transfer on SAR-RARP50, which has no 3D labels. We render the estimated instrument silhouette and score its overlap with the annotated tool masks, a 2D proxy that still penalizes wrong depth through scale and position.

Experiment 5 is the tiny-feature ablation. It compares geometry-guided native-resolution crops, LIP pooling, plain downsampling, and full-resolution processing. Tip error is binned by apparent tip size in pixels, with latency reported for each variant.

The proposal is falsified in two cases. If the no-anchor transformer matches the anchored model on occlusions longer than its window, the anchor buys nothing. If the anchor's own error dominates on manually held laparoscopic footage, where the pivot is less rigid, the fixed-pivot assumption is too strong for that setting. Either outcome would redirect the work.