The Patient's Own Prostate as a Ruler: Metric Laparoscopic Depth from Preoperative Biparametric MRI

A single laparoscope camera cannot tell a small nearby structure from a large distant one. Depth networks trained on such video return "relative" depth, correct in shape but off by an unknown scale factor that can change from frame to frame. In robot-assisted prostate removal, the current fix is to measure a surgical tool of known shaft radius and rescale the depth map to millimeters from it. That fix breaks whenever the tools leave the view, stop moving, or are hidden by blood or smoke. Before surgery, however, nearly every such patient has already had a prostate MRI. That scan measures the gland in real millimeters. The contrast-free version, called biparametric MRI (bpMRI), uses only anatomical T2-weighted and diffusion-weighted images, and it is increasingly the default scan.

This note proposes using the patient's bpMRI-derived gland surface as a second, always-present ruler. We fit the monocular relative depth map directly against a rendered depth image of the patient-specific gland mesh, and we treat both the camera pose and the depth scale as unknowns. We then combine this organ anchor with the instrument anchor whenever tools are visible. The claim is narrow and testable: during tool-free intervals, an organ anchor calibrated from the patient's own bpMRI keeps metric depth error close to the tool-present level, while simpler strategies drift or collapse. The fusion also tells the system which anchor to trust at each moment.

What Exists and What Is Missing

Two papers bracket the idea. Enhanced Scale-Aware Depth Estimation for Monocular Endoscopic Scenes with Geometric Modeling (Wei et al.) supplies the method we build on. It models da Vinci instrument shafts as cylinders of known radius, recovers their 3D pose from image contours, and fits a linear scale and offset that turns Monodepth2-style relative depth into millimeters. It reports an MAE of 6.347 mm, an RMSE of 8.637 mm and δ<1.25 accuracy of 88.0% on 95 prostatectomy clips. Its stated limitation motivates this proposal: without visible, unoccluded shafts there is no scale at all.

An augmented reality overlay for navigated prostatectomy using fiducial-free 2D-3D registration (Bender et al.) registers an MRI-derived prostate model to monocular video by differentiable rendering, which means optimizing the pose through a renderer so that a rendered image matches the observed one. Its matching signal is surgeon-drawn 2D strokes tracked with CoTracker. It reports a median target registration error of 6.11 mm on synthetic data, and it reduces median translation error from 4.86 to 3.04 mm and rotation error from 3.66° to 1.43° relative to PnPRansac. This work is the closest to ours and also our strongest competitor. Registering a known-size model with known camera intrinsics already fixes metric scale implicitly. Our proposal differs in three concrete ways. We use dense relative depth rather than sparse drawn strokes as the matching signal. We make scale an explicit free variable instead of trusting the rigid model's size, so that deformation and segmentation error show up as a measurable scale residual rather than a silent bias. And we fuse the organ anchor with the instrument anchor, with neither the manual initialization nor the stroke-drawing step Bender's method requires.

The bpMRI literature constrains what the anchor may be. Evaluating deep learning and radiologist performance in volumetric prostate cancer analysis with biparametric MRI and histopathologically mapped slides (Yılmaz et al.) found that a 3D U-Net underestimates tumor volume badly against whole-mount histopathology: a median of 1.2 mL at a 50% threshold and 0.7 mL at 75%, versus 2.3 mL in pathology. Radiologists came closer at 2.1 mL. We therefore anchor scale to the gland surface only, never to tumor contours. Tumor overlays are treated as a downstream consumer whose known shrinkage must not feed back into the scale estimate. The noninferiority evidence for bpMRI (Twilt et al., Li et al.) is why we design around bpMRI rather than contrast-enhanced multiparametric MRI (mpMRI). If the anchor works from the cheaper scan, it works for the growing population that receives only that scan.

Core Mechanism

For each frame, a relative depth network outputs $d(p)$ at pixel $p$. The bpMRI gland mesh, segmented from T2-weighted images at the scan's physical voxel spacing, renders a metric depth image $D(p;T)$ under camera pose $T$ with the laparoscope's known intrinsics. We solve

$$\min_{s,b,T}\ \sum_{p\in\Omega_g}\rho\big(s\,d(p)+b-D(p;T)\big)\;+\;\lambda\sum_{q\in\Omega_i}\rho\big(s\,d(q)+b-Z_i(q)\big),$$

where $s$ and $b$ are the depth scale and offset, $\Omega_g$ is the set of pixels segmented as exposed prostate surface, and $\rho$ is a robust penalty. The second term is Wei et al.'s instrument constraint: $\Omega_i$ is the set of instrument pixels, $Z_i$ their depths from cylinder pose, and $\lambda$ is set to zero when no tools are detected. $T$ is initialized from the previous frame's solution, and the first frame is initialized by a coarse search. The mesh is rigid, but $s$ is estimated per frame. Two diagnostics come out of the fit. One is a curvature-based identifiability score that measures how strongly the visible surface patch constrains scale; a nearly flat patch leaves scale and distance entangled, the planar ambiguity Bender et al. discuss. The other is the residual, which tells us when deformation has made the MRI shape unreliable. When both anchors are available, each frame's weighting between them follows these diagnostics.

Research Program

The program has three parts. First, we build an evaluation set with paired preoperative bpMRI and stereo da Vinci video. Following Wei et al.'s protocol, stereo matching supplies ground-truth depth and one channel is used as the monocular input. We create tool-free test conditions in two ways: by selecting naturally tool-absent intervals, and by masking instruments out of tool-present frames. Masking lets us compare the same frames with and without the instrument anchor.

Second, because paired clinical data will be small, we build a synthetic companion in the spirit of Bender et al.'s 504 synthetic scenes. We segment glands from the T2-weighted volumes in FastMRI Prostate (Tibrewala et al., 312 patients), apply controlled non-rigid deformations and partial exposure, and render laparoscopic views with exact depth. This lets us sweep the two quantities we expect to govern success: how much gland surface is visible, and how far the tissue has deformed from its MRI shape.

Third, we analyze failure. We will characterize when the organ anchor is worse than holding the last known scale. The accompanying diagnostic is intended as a practical output, a per-frame confidence a navigation system can display.

Baselines

The naive reference point holds the last instrument-derived scale and offset fixed through a tool-free interval. It costs nothing, and if the relative depth network's scale is stable over short intervals, it may be hard to beat early on. We run it on exactly the same frames and report error as a function of seconds since the tools disappeared. Beating it means lower MAE at every horizon beyond a few seconds, without being worse at short horizons.

The strongest existing method for tool-present frames is Wei et al.'s instrument-geometry scaling, reimplemented with the same relative depth backbone as ours. On their in-house data it reports MAE 6.347 mm, RMSE 8.637 mm and δ<1.25 of 88.0%. Our data differ, so these numbers serve as a sanity target for our reimplementation, not as a direct comparison. On tool-free frames it has no output, and we score it by its hold-last fallback. Beating it means matching its tool-present error with fusion enabled, and staying within a pre-registered margin of its tool-present error when tools are masked. We fix that margin at 20% relative MAE before running.

The strongest registration competitor is Bender et al.'s differentiable-rendering 2D-3D registration. We give it the same gland mesh and intrinsics, and we read its metric depth off the rendered registered model. To keep the comparison fair, it receives the same initialization as our method rather than manual alignment, and synthetic stroke cues where no surgeon-drawn cues exist. Its reported median target registration error of 6.11 mm on synthetic data is the target figure. Beating it means lower depth MAE, together with a scale estimate that degrades gracefully rather than abruptly as deformation increases.

We also run Metric3D v2 (Zhang et al.) zero-shot with its canonical camera transform and the known endoscope intrinsics. It reports an absolute relative error of 0.044 on KITTI and 0.047 on NYUv2. We found no reported result on Lacuna for Metric3D v2 on endoscopic prostatectomy video.

The ablation that isolates our key ingredient replaces the patient-specific bpMRI mesh with a population-mean prostate shape of average size. We also run a version with the patient mesh deliberately rescaled by ±10%. If the mean shape does as well as the patient's own scan, the MRI contributes nothing beyond a generic size prior.

All methods share the same relative depth backbone, fine-tuning data and tuning budget.

Why not just…?

Why not just use the stereo endoscope? On a da Vinci it gives depth directly, and we use it as ground truth for exactly that reason. Wei et al. and Bender et al. both note, however, that single-port, flexible and standard monocular laparoscopes cannot carry stereo optics, and that stereo matching leaves holes on specular, featureless capsule tissue. The method targets those monocular platforms. The stereo-equipped robot serves only as the measuring instrument for validation.

Why not just rely on instrument geometry? When tools are visible, it is excellent, and our fusion keeps it. The question is only what happens without tools. The tool-masking experiment answers this directly by comparing the same frames with and without the instrument anchor.

Why not just register the MRI model as Bender et al. do and render its depth? This is the real threat, because a registered known-size model already carries metric scale. If full registration on dense depth matches our explicit-scale fit, the contribution shrinks to replacing drawn strokes with dense depth. That would still be useful, since it removes the manual steps, but it is a smaller claim. The deformation sweep settles the question. We expect a rigid registration to absorb deformation into a wrong distance, while a free scale with a residual check exposes it. If that expectation fails, we will report it.

Why not just use a metric depth foundation model such as Metric3D v2? Its metric scale comes from priors learned on everyday scenes. Wet, specular, deforming tissue is far outside that training data. The zero-shot baseline measures this directly. If it proves competitive, the organ anchor should then be tested as a correction applied on top of it.

Why not just use full mpMRI for a better model? The anchor uses the gland boundary, which T2-weighted images delineate. Contrast enhancement mainly informs lesion assessment, not gland shape. Yılmaz et al.'s radiologists came within 0.2 mL of pathology tumor volume on bpMRI, which suggests the structural boundaries are present in these sequences. Tumor-margin shrinkage is a separate problem that this work deliberately keeps out of the scale estimate.

Decisive Checks

The idea succeeds if three things hold. On masked tool-free frames of the clinical set, fused organ-anchored depth stays within 20% relative MAE of tool-present instrument scaling. It beats hold-last scale beyond short horizons. And the patient-specific mesh beats the population-mean mesh by a margin larger than the run-to-run variance.

It fails informatively if the synthetic sweeps show that scale becomes unidentifiable below some fraction of visible gland surface, or above some deformation level, that is typical of real surgery. In that case, the identifiability score becomes the main deliverable: a reliable signal of when monocular navigation should not display metric distances.