Tracking the Endoscope Against Pre-Operative Geometry, Not Appearance

An endoscope camera inside the body has no GPS. It sees wet, textureless tissue lit by a lamp that moves with it. Water, debris and smoke stick to its lens, and the organs flex. Camera-tracking methods that work only from the video drift over time, and in surgery drift means the navigation overlay points at the wrong millimeter of anatomy. Self-driving cars face the same problem in tunnels and parking garages. They solve it by matching live sensor data to a prior map, which turns unbounded error build-up into a bounded alignment problem. Surgery has a prior map too: the patient's pre-operative computed tomography (CT) scan. Yet no method in the literature we reviewed uses it for continuous, real-time camera tracking.

We propose to track the endoscope by aligning predicted depth, not image intensity, to the pre-operative surface on every frame. A reliability gate throws out pixels corrupted by contaminants stuck to the lens. The bet rests on an asymmetry that the colonoscopy benchmark SimCol3D exposed. Single-image depth prediction is close to solved there: the winning team reached 0.03 cm L1 on synthetic colons. Camera pose estimation, by contrast, drifted badly on real videos. Appearance is the channel that lighting and lens contamination destroy. Geometry is the channel that deep networks now recover well. So we localize using geometry against a geometric map, and never ask the map to explain pixel colors.

Core Claim

Consider per-frame alignment of rescaled monocular depth to a distance field built from the pre-operative surface, combined with a gate that detects image-locked lens contaminants. We claim this method bounds trajectory error over long sequences and survives optical degradation. On degraded colonoscopy sequences it should match or beat photometric SLAM systems that model lens contamination explicitly, while running at video rate. We also claim that its accuracy degrades smoothly and measurably as the pre-operative map disagrees with the intraoperative anatomy. That second claim matters, because it tells clinicians how far such a map can be trusted.

The driving literature supplies the design lesson. What is the best grid-map for self-driving cars localization? compared map types. The most precise maps (LiDAR reflectivity) were less stable under illumination and seasonal change than coarser semantic maps. Vehicle Localization and Control on Roads with Prior Grade Map showed that even a one-dimensional road-grade profile bounds along-track drift. LHMap-loc kept only the most salient 20% of LiDAR points and still localized a monocular camera at 0.03 m and 0.33° on KITTI at about 77 frames per second. The common thread is that the right map keeps the invariant structure and discards appearance. In an endoscopic cavity, the invariant structure is surface shape.

Relation to Prior Work

The closest work is Self-P2IR. It lifts single laparoscopic keyframes to 3D with a monocular depth model and registers them rigidly, then non-rigidly, to a pre-operative liver model. It reports 78.89% Dice and 2.97 mm Chamfer distance on P2I-LReg. We borrow its core move of registering predicted depth to a pre-operative model, but we change what it is used for. Self-P2IR registers isolated keyframes and fixes metric scale with an offline PCA step. It never tracks a trajectory. We estimate scale jointly with pose on every frame, carry the pose forward in time, and evaluate trajectory error.

Towards navigation in endoscopic kidney surgery motivates the problem from the opposite side. It registered a structure-from-motion reconstruction to a CT mesh with sub-millimeter inlier error. However, the pipeline took about eight hours and needed manual scale adjustment because automatic global registration failed. We avoid global registration on every frame: each frame starts from the previous pose, and global search is reserved for recovery after the camera goes blind.

Three intraoperative SLAM systems motivate and constrain the design. EndoFlow-SLAM is a real-time Gaussian-splatting system that adds a per-Gaussian flow constraint for breathing motion. It reaches 0.23 mm ATE (absolute trajectory error) on the static C3VD phantom but 15.47 mm on the breathing StereoMIS sequences, which is evidence that map-free tracking drifts once the scene moves. EndoMD-SLAM separates a static anatomy field from a per-view transient field for lens contamination, and relocalizes from a memory of clean keyframes. It cuts ATE on degraded C3VDv2 from about 35 mm to 3.02 mm, but runs at 0.95 FPS. Its result constrains us: ignoring lens contamination is fatal, so we must handle it. Our handling is a cheap gate rather than a second radiance field. NFL-BA shows that photometric bundle adjustment fails under a moving co-axial light unless inverse-square shading is modeled, and that its correction still assumes diffuse surfaces. That limitation is part of our argument for dropping photometric matching against the map entirely. Finally, GS-EVT tracks a camera by rendering a pre-built splatting map and comparing appearance. It is the natural "track against a rendered prior" design, and we test its appearance-based variant as an ablation.

Research Program

The tracker keeps a pose $T_t$ and a depth scale $s_t$ for frame $t$. A pretrained monocular depth network produces a relative depth map $D_t$. A distance field $\phi_M$ is precomputed once from the segmented pre-operative surface $M$; it returns, for any 3D point, the distance to the nearest point of $M$. The per-frame estimate solves

$$ (T_t, s_t) = \arg\min_{T,s} \sum_{p} w_t(p)\, \rho\!\left(\phi_M\!\left(T\, \pi^{-1}(p,\, s\, D_t(p))\right)\right) + \lambda\, d\!\left(T,\; T_{t-1}\Delta_t\right)^2 . $$

Here $\pi^{-1}(p, z)$ back-projects pixel $p$ to depth $z$ using the calibrated camera. The function $\rho$ is a truncated robust penalty, so tissue that has moved far from the pre-operative shape stops pulling on the pose. $w_t(p)\in[0,1]$ is the reliability gate. $\Delta_t$ is a short-range motion estimate from frame-to-frame optical flow. $d$ is a pose distance weighted by $\lambda$, which keeps the estimate close to the odometry prediction when the map term is weak.

Distance-field lookups are cheap and the pose starts from the previous frame, so a few Gauss–Newton iterations should suffice. The target is at least 20 FPS, the real-time threshold that EndoMD-SLAM misses. We will report whether we reach it.

The gate exploits one physical fact that the file highlights. Contamination on the lens stays locked in image coordinates while the camera moves. A pixel is down-weighted when two things hold at once: its residual to the map is large, and its image content persists at zero displacement while the flow elsewhere in the frame is clearly non-zero. When the gated area exceeds a threshold, the tracker declares itself blind. It then propagates the pose with constant velocity and stops trusting the map term. When enough clean pixels return, it runs a small multi-hypothesis search around the propagated pose, minimizing the same distance-field objective, before resuming normal tracking.

The pre-operative map will never exactly equal the intraoperative anatomy. We therefore treat the size of the mismatch as an experimental variable rather than assuming it away.

Proposed Experiments

The primary testbed is C3VD and its degraded split C3VDv2. These phantom sequences come with ground-truth camera poses, and C3VD provides the ground-truth 3D phantom topography. That topography serves as the "pre-operative" surface. C3VDv2 carries explicit water-on-lens and debris-on-lens tags; we need to confirm that its sequences share usable phantom geometry. The SimCol3D synthetic colons, including the unseen Colon III, provide longer trajectories with known meshes for testing drift over sequence length.

Using the true surface as the map is optimistic. We therefore evaluate against deliberately corrupted maps:

  • rigid misregistration of the map by a few millimeters and degrees;
  • smooth, spatially varying deformations with peak displacements swept from zero to roughly one centimeter;
  • downsampling of the map to CT-like voxel resolution.

This produces a curve of ATE against map error, which we consider as important as the headline number.

For clinical realism, the kidney CT-and-video data and P2I-LReg are the only sources in the file that pair real anatomy with pre-operative models. Neither provides continuous ground-truth trajectories. We would use them only for qualitative checks and for P2I-LReg-style per-frame Dice, and we will say so plainly.

Metrics are ATE and relative pose error in millimeters, the fraction of frames tracked within 5 mm, recovery time after a blinding event, and frames per second on a single GPU.

Baselines

The naive reference is frame-to-frame unsupervised visual odometry, Endo-SfMLearner. It uses attention toward textured regions and a brightness-aware photometric loss, and it has no map. We run it with its released training recipe on the same sequences. It shows how much drift remains to be bounded, and any map-anchored method should beat it on every long sequence.

The strongest existing method on degraded sequences is EndoMD-SLAM, which reports 3.02 mm ATE on degraded C3VDv2 at 0.95 FPS. We run it on the identical C3VDv2 sequences with its published settings. Beating it means matching or lowering 3.02 mm ATE on those splits while running at least ten times faster. Matching its accuracy at video rate would already be a meaningful outcome.

On clean and dynamic data the reference is EndoFlow-SLAM, which reports 0.23 mm ATE on C3VD and 15.47 mm on StereoMIS. We do not expect to beat 0.23 mm on short clean phantom sequences. The test is whether our error stays flat as SimCol3D sequences lengthen while its error grows. StereoMIS has no pre-operative surface in the file, so we cannot run our method there. We note this as a limitation rather than inventing a comparison.

We add a loosely coupled baseline. It runs EndoFlow-SLAM, and every $k$ frames performs a Self-P2IR-style registration of the current depth to the same corrupted map, resetting the pose. Every method gets the same depth network, the same corrupted maps, and an equal hyperparameter search budget.

Three ablations isolate the key ingredients. The first removes the gate and keeps only the robust penalty. The second replaces the depth-to-distance-field term with photometric matching against a rendering of the map textured with NFL-BA-style inverse-square shading, which tests whether appearance is actually worse than geometry. The third sets $\lambda$ very large, so the map term has almost no influence.

Why not just…?

Why not just use map-free SLAM, which already reaches 0.23 mm on C3VD? That figure comes from short, static phantom sequences. The same system reports 15.47 mm on breathing tissue. SimCol3D found severe pose drift on real colonoscopy even for teams with near-perfect depth. Map-free error has no mechanism that bounds it over time. Our sequence-length experiment on SimCol3D shows directly whether that matters at procedure-relevant durations. If map-free error stays flat on long sequences, our motivation weakens, and we will report that.

Why not just run SLAM and periodically register to CT? This is the real threat, since it reuses existing parts. Its weaknesses are the pose jumps at each reset and the frames between resets that remain unanchored during a lens flush. It may also need global registration, which failed without manual help in the kidney study. The loosely coupled baseline above settles the question across several values of $k$. If it matches per-frame coupling at equal speed, the contribution narrows to the gate and the map-error analysis.

Why not just use a robust loss and skip the gate? EndoMD-SLAM showed that unmodeled lens debris gets baked into the map as floating geometry and drives drift to about 35 mm. A truncated penalty helps against scattered outliers, but a contaminant covering half the frame for many frames is not scattered. The no-gate ablation on the water-on-lens splits measures the difference.

Why not just track against a photorealistic rendering of the map, as GS-EVT does? A CT surface carries no tissue color. Even if it did, NFL-BA shows that moving near-field light and specular wet tissue break photometric matching unless shading is modeled, and modeled only under diffuse assumptions. The photometric-map ablation tests this directly. If that ablation matches depth alignment on C3VDv2, our central claim that geometry is the more reliable channel is wrong.

Decisive Checks

The idea succeeds if three things hold. First, on degraded C3VDv2 it tracks within the error EndoMD-SLAM reports while running at video rate. Second, on long SimCol3D trajectories its error stays roughly constant while map-free baselines grow. Third, the ATE-versus-map-error curve shows millimeter-level tracking under several millimeters of prior deformation. The idea fails if tracking breaks down at deformations smaller than those expected between CT and surgery. It also fails if the loosely coupled baseline or the photometric ablation performs as well; either outcome would say that the per-frame geometric coupling is not what does the work.