A single endoscope camera cannot tell a 2 mm polyp at close range from a 2 cm polyp farther away. Neural depth networks give good relative depth, meaning which point is nearer than which, but not distances in millimeters. The most direct fix in robotic surgery uses the instrument as a ruler: its shaft is a cylinder of known radius, so its 3D pose, and hence a metric scale for the scene, can be computed from its outline. Enhanced Scale-aware Depth Estimation for Monocular Endoscopic Scenes with Geometric Modeling does exactly this. That ruler disappears whenever the tool is retracted, occluded by tissue, or covered in blood. The method also uses it to fit one global scale for the whole image, even though depth networks distort space unevenly across a scene. The polyp-reporting work in From Generalist to Specialist names the consequence directly: without a physical reference in view, monocular sizing stays ill-posed.
We propose to treat metric scale as a quantity that varies smoothly over the image and persists over time. When the tool is visible, its metric 3D points calibrate a low-dimensional scale field, a per-pixel multiplier that turns relative depth into millimeters, instead of a single scalar. Those calibrated tissue points are written into a short-horizon metric memory of the surrounding anatomy. When the tool leaves, each new frame's relative depth is aligned to the remembered metric tissue where the two views overlap, so the tissue itself carries the scale forward. The lesson we borrow from the volumetric-analysis literature is a narrow one: collapsing a spatially varying quantity to one summary number discards information that matters. That is what full-volume versus single-slice radiomics found for tumors in A modified method for CT radiomics region-of-interest segmentation, and what 3D versus plane-parallel cloud models found in Multiple-Scattering Microphysics Tomography. Here the quantity is depth scale rather than tissue texture. We do not claim to reconstruct tissue interiors from a camera.
Core Claim
Two claims are testable and separable. First, in endoscopic scenes a spatially varying scale field fitted to tool anchors gives lower metric depth error than the global scale-and-shift fit used by the instrument-anchoring method. This is the endoscopic analogue of the 30% inverse-RMSE gain that Monocular Visual-Inertial Depth Estimation reports for its dense ScaleMapLearner over global alignment on VOID. Second, and more important, propagating scale through a metric tissue memory keeps error low for many frames after the anchor vanishes. Without it, current methods fall back to uncalibrated relative depth or to whatever scale prior a supervised network has absorbed for one particular endoscope. We will summarize the second claim with a "scale half-life": the number of anchor-free frames before metric error doubles. If the first claim fails, meaning the oracle headroom of a spatial field over a global fit is negligible on endoscopic data, we drop the field and keep only the memory component.
Research Program
The base relative depth comes from a frozen foundation model adapted to surgery. We will use the LoRA-adapted DINOv2 recipe of Surgical-DINO, retrained to output affine-invariant depth so that scale is handled entirely downstream. Metric anchors come from the cylinder-and-Plücker-coordinate tool pose solver of the instrument-anchoring paper, which gives 3D points along the shaft and at the tip.
The scale field is deliberately low-capacity, so that a handful of anchors concentrated on the tool can constrain it without overfitting. For a pixel $u$ with relative depth $r(u)$, metric depth is modeled as
$$\hat Z(u) = \exp\Big(\textstyle\sum_{k=1}^{K} w_k\,\phi_k(u)\Big)\, r(u) + t,$$
where the $\phi_k$ are $K$ smooth basis functions over the image (low-order splines, $K$ around 6 to 16), the $w_k$ are per-frame weights, and $t$ is a shift. The weights and shift are solved by robust least squares against tool anchors and memory points, with a smoothness penalty that pulls the field toward a constant wherever data are sparse. This differs from ScaleMapLearner, which learns an unconstrained per-pixel map from a VIO scaffold. We choose the explicit low-dimensional form because tool anchors are spatially clustered, not scattered across the scene as VIO features are, and an unconstrained map would extrapolate freely into tissue regions with no support.
The memory is a keyframe buffer of metric tissue points with appearance descriptors, taken from the same frozen backbone. Frame-to-frame camera motion is estimated from descriptor correspondences on tissue only, with tool regions masked. Points whose implied position disagrees with their neighbors beyond a threshold are down-weighted, which tolerates breathing and retraction motion without a full deformation model. Each memory point also carries a confidence that decays with time since it was last observed. This decay is what makes the scale half-life a meaningful quantity rather than an artifact. Learning in the system is limited to the depth adapter; alignment is optimization over at most a few dozen variables per frame, so the 66 ms per frame of the anchoring method is a realistic latency target.
Baselines
The naive reference point has two arms. One is plain relative depth with no metric recovery, scored only on scale-invariant metrics. The other is an "anchor-hold" baseline that, after the tool disappears, aligns each frame's relative depth to the previous frame's metric depth with one global scalar fitted on tissue overlap. Anchor-hold is the cheapest form of temporal propagation, and our memory has to beat it.
The strongest instrument-anchored competitor is the global fit of Enhanced Scale-aware Depth Estimation. It reports 6.347 mm MAE, 8.637 mm RMSE and 88.0% δ<1.25 on its in-house da Vinci prostatectomy data. That data is not public, so we will reimplement the global scale fit on top of the same relative depth backbone and the same anchor points we use. Only the alignment model then differs between the two methods.
The strongest dense-refinement competitor is ScaleMapLearner from Wofk et al., trained with tool anchors in place of VIO points. It gets the same training sequences and a matched hyperparameter search budget.
The strongest supervised metric competitor is Surgical-DINO itself, which reports Abs Rel 0.053 on SCARED against 0.059 for AF-SfMLearner. We will confirm whether that number is computed after per-frame median rescaling. Our headline metrics use no ground-truth rescaling, so if it is, we will rerun Surgical-DINO under our protocol rather than quote it.
We also include zero-shot Metric3D v2, with calibrated endoscope intrinsics fed through its canonical camera transform. No reported result was found on Lacuna for Metric3D v2 on SCARED or Hamlyn.
Two ablations isolate the key ingredients: the field with no memory, and the memory with a global scale only. Beating the baselines means lower anchor-free metric MAE than every competitor at matched latency, with the gain persisting at 30 or more frames after anchor loss, and no regression while the tool is visible.
Why not just…?
Why not just fit one global scale whenever the tool is visible, and hold it otherwise? This is the anchor-hold baseline, and it is a real threat. If endoscopic depth networks distort little and the camera moves slowly, a held global scale may be nearly as good as our method. The oracle experiment below measures directly how much a spatial field could gain over a global fit when both are given dense ground truth, and the anchor-dropout curve shows whether holding a single scalar drifts faster than a memory of many tissue points. If both gaps are within noise, the honest conclusion is that the simpler method suffices, and we will report it.
Why not just train a supervised metric network such as Surgical-DINO on surgical data? A network trained on one endoscope learns that endoscope's typical working distance as a prior. That works in-domain, which the 0.053 Abs Rel on SCARED suggests, but nothing in its output ties the scale to physical evidence in the current frame. The cross-camera test on Hamlyn and the zoom-perturbation test below are designed to expose that prior. If Surgical-DINO's metric error stays flat under those shifts, our claim that physical anchoring is needed is weakened.
Why not just use the camera intrinsics with Metric3D v2? In robotic endoscopy the intrinsics are usually calibrated, so the canonical camera transform is well posed. The file does give a reason for doubt: Surgical-DINO reports that natural-image foundation models degrade sharply on surgical anatomy. But that evidence concerns depth quality, not scale specifically, so we run Metric3D v2 as a full baseline rather than dismiss it.
Why not just run monocular SLAM or structure-from-motion and anchor it once? Monocular reconstruction has no metric scale of its own. It can propagate scale but needs an anchor, which is the same structure as our proposal. The instrument-anchoring paper argues that structure-from-motion breaks on deforming, specular tissue, which is why we use a short decaying memory with outlier down-weighting instead of a global map. The memory-horizon sweep tests whether a longer, SLAM-like memory helps or hurts.
Proposed Experiments
The first decisive check is headroom. On SCARED, using stereo ground truth, we fit the oracle global affine model and the oracle $K$-basis scale field to all valid pixels of each frame and compare the residual metric error. The question is whether endoscopic depth distortion is spatially structured enough to matter. This check costs a day and decides whether the field component stays in the proposal.
The second check is anchor dropout. SCARED does not provide tool pose labels in a form we can rely on, so we use two complementary sources. In SurRoL, which provides ground-truth instrument poses and from which we can render depth, we take genuine tool anchors and remove them on a schedule. On SCARED, we use ground-truth depth sampled along a thin synthetic shaft-shaped region as a proxy anchor, disclosed as such. For both, we plot metric MAE in millimeters and Abs Rel without rescaling against frames since the last anchor, and we report the scale half-life for every method in the Baselines section.
The third check is camera shift. All methods are calibrated on SCARED and evaluated zero-shot on Hamlyn. We also apply synthetic zoom, by cropping and resizing with intrinsics updated for methods that accept them, to SCARED test frames. This checks whether gains come from physical anchoring rather than a learned distance prior.
Finally, as a downstream sanity check, we measure the length of short structures in millimeters (vessel segments and marked tissue spans with stereo-derived ground truth), with and without a visible tool. Lesion sizing is the clinical need that From Generalist to Specialist identified, where its retrieval-based system reaches 0.200 cm diameter MAE on EndoVL without recovering geometry. EndoVL lacks depth ground truth, so we will not claim a head-to-head comparison there.
References
- Enhanced Scale-aware Depth Estimation for Monocular Endoscopic Scenes with Geometric Modeling
- From Generalist to Specialist: A Context-Fusion Framework for Endoscopic Polyp Reporting with a Frozen VLM
- A modified method for CT radiomics region-of-interest segmentation in adrenal lipid-poor adenomas: a two-institution comparative study
- Multiple-Scattering Microphysics Tomography
- Monocular Visual-Inertial Depth Estimation
- Surgical-DINO: adapter learning of foundation models for depth estimation in endoscopic surgery
- Metric3D v2: A Versatile Monocular Geometric Foundation Model for Zero-shot Metric Depth and Surface Normal Estimation