Self-Calibrating Near-Field Shading Refinement for Monocular Endoscopic Depth

A modern depth network can look at a single endoscope frame and guess the shape of the tissue. That guess has three standard defects. It is correct only up to an unknown scale and offset. It does not know the camera's focal length. And it smooths away fine surface detail such as folds, vessels and small bumps. Endoscopy also offers a physical cue that most fast pipelines discard. The only light comes from a lamp at the tip of the scope, next to the lens, so the brightness of each pixel falls off roughly with the inverse square of its distance and with the angle between the surface and the lamp. We propose a per-frame GPU solver that runs in under 50 ms. It takes the network's relative depth and fits it to this lighting physics. In one fitting step it recovers the depth offset, the focal length, a sequence-constant scale, and high-frequency surface detail, with no camera calibration and no depth sensor.

The two-sentence version is this. A light attached to the camera makes image brightness a known function of 3D geometry, and that function depends on depth offset and focal length in ways the network's guess does not fix. Fitting the guess to the observed brightness in real time therefore calibrates the camera and sharpens the surface simultaneously. The capability matters in practice. Current online surgical reconstruction either assumes calibrated intrinsics or uses raw foundation-model depth with no physics. Classic real-time shading refinement needs an active depth sensor and assumes distant light.

Core Claim

Each existing line of work covers part of the problem. Real-time shape-from-shading exists: Real-time shading-based refinement for consumer depth cameras refines depth at 15 to 93 FPS with a patch-based GPU Gauss-Newton solver. It needs a Kinect-class metric depth anchor, known intrinsics, and distant spherical-harmonic lighting, all of which are wrong for an endoscope. Near-field lighting has been shown to be a strong geometric cue. NFL-BA adds an inverse-square shading model to SLAM bundle adjustment, and PC-Depth bakes spotlight falloff and specular-highlight constraints into the training losses of a depth network. Both still assume known intrinsics and neither runs as a streaming refinement stage. Calibration-free streaming exists as well. StreamSplat absorbs the unknown camera into an orthographic canonical space, and it admits this fails for close-ups with strong perspective. Endo3R uses a DUSt3R-style model at 19–20 FPS. Neither uses shading at all.

Our claim is narrower and testable. For a camera with a co-located near-field light, a single frame's shading constrains the depth offset and the focal length that the network leaves free. A few warm-started Gauss-Newton iterations per frame can recover both, together with surface detail, inside a real-time budget. The closest conceptual precedent is Novel Depth Cues from Uncalibrated Near-field Lighting. It extracts depth ordering from an uncalibrated moving light, but it needs a stationary camera and many frames, and it runs offline. We use the opposite configuration: a moving camera with its light rigidly attached, one frame at a time.

What Shading Can and Cannot Identify

We write the image formation model as

$$I(p) = g\,\rho(p)\,\mu(p)\,\frac{\max\big(0,\ n(p)\cdot \hat{\ell}(p)\big)}{\|X(p)\|^2}, \qquad X(p) = z(p)\,K_f^{-1}[p;1], \qquad z(p) = s\,d(p) + t + \delta(p).$$

The symbols are as follows:

  • $I(p)$ is the linearized intensity at pixel $p$.
  • $g$ is the unknown light gain.
  • $\rho(p)$ is the surface albedo.
  • $\mu(p)$ is a smooth radial term that absorbs the spotlight profile and vignetting.
  • $n(p)$ is the surface normal, computed from depth gradients.
  • $\hat{\ell}(p) = -X(p)/\|X(p)\|$ is the direction to a light co-located with the camera.
  • $K_f$ is the intrinsic matrix with unknown focal length $f$, with the principal point initialized at the image center.
  • $d(p)$ is the network's relative depth.
  • $s, t$ are the unknown scale and shift.
  • $\delta(p)$ is a high-frequency residual depth.

Writing the model out shows the limits honestly. If the whole scene is multiplied by a factor, normals are unchanged and intensity drops by the square of that factor. That loss is exactly absorbed by $g$, so absolute metric scale is not observable from shading alone. Two quantities are observable. The shift $t$ changes the ratios of distances across the image, which the falloff term exposes. The focal length $f$ changes both the ray directions and the normals recovered from depth gradients, and therefore changes the predicted $n\cdot\hat{\ell}$ pattern.

The gain $g$ is fixed for a given scope and light setting. Fitting it once per sequence therefore makes the scale consistent over time, even though the scale itself is not metric. A single known light power, or one external measurement, would make it metric. We will report scale-aligned metrics as the field does, and in addition report how much the recovered scale varies over time.

Research Program

The system is built in four stages.

Backbone. A fast relative-depth network provides $d(p)$. We use the Surgical-DINO configuration, LoRA-adapted DINOv2, because it runs at 18.2 ms per frame on an RTX 3090 according to Surgical-DINO. That leaves roughly 30 ms for refinement within a 50 ms budget.

Refinement solver. We adapt the patch-based Gauss-Newton solver of Real-time shading-based refinement for consumer depth cameras. Its distant spherical-harmonic lighting is replaced by the near-field model above. Its sensor depth anchor is replaced by the network's prediction, softly weighted for the low frequencies. Global unknowns ($s, t, f, g$ and a few coefficients for $\mu$) are optimized in an outer step, and the per-pixel residual $\delta$ is updated in patches. Each frame warm-starts from the previous frame's global parameters, and $f$ and $g$ are updated with a slow running average. This is what should keep the per-frame cost to a few iterations.

Albedo. Albedo is handled the same way the RGB-D solver handles texture copying: we downweight shading residuals across chromaticity edges, such as vessels, so that color changes are not mistaken for slopes.

Specular highlights. Highlights are detected and removed from the diffuse term. Following PC-Depth, we reuse them as anchors where the normal must point back toward the co-located camera and light. This turns a known failure mode of NFL-BA into a constraint.

The analysis component has three parts. First, a sensitivity study of the shading residual with respect to $f$ and $t$ as a function of scene depth range and relief. We expect focal length to be poorly constrained on flat, fronto-parallel tissue and well constrained in tubular colon views. Mapping where self-calibration works is itself a result. Second, a characterization of when $\mu$ and the falloff term can be separated. Because depth varies independently of image radius across frames, a sequence-level $\mu$ should be identifiable. Third, a latency profile per stage.

Baselines

Do-nothing reference. The backbone's raw relative depth, aligned to ground truth by per-frame median scaling, run with the same intrinsics-free input. Any refinement must beat this on depth error, on normal error, and above all on high-frequency structure. Otherwise the solver is decoration.

Strongest learned competitor in the physics direction. PC-Depth uses near-field photometric losses during training and reports AbsRel 0.076 on C3VD and AbsRel 0.104 zero-shot on SERV-CT when trained on C3VD. It needs known intrinsics, which we will supply to it. That advantage favors the baseline. Beating it would mean matching or lowering AbsRel on C3VD with no intrinsics given to our method, and lowering normal angular error.

Strongest real-time depth competitors. On SCARED, Surgical-DINO reports AbsRel 0.053. On C3VD, EndoStreamDepth reports δ1 of 0.952 at about 24 FPS on an A6000. We will retrain or reuse both on the same splits and time all methods on one RTX 3090.

Calibration-free competitor. Endo3R reports AbsRel 0.124 on SCARED at 19–20 FPS. For focal-length recovery, no reported result was found on Lacuna for focal-length error of any calibration-free endoscopic method. We will therefore extract Endo3R's implied intrinsics ourselves and compare them against the dataset calibration.

Ablations that isolate the key ingredient. The central ablation keeps the solver, schedule and budget fixed and swaps only the lighting model, replacing near-field falloff with distant spherical harmonics as in the original RGB-D solver. If the spherical-harmonic version refines detail equally well and also recovers $f$ and $t$, our central claim is wrong. A second ablation freezes $f$ at a wrong value (±20%) to measure how much self-calibration matters. A third removes the specular anchor.

All learned baselines receive the same training data and a matched tuning budget, defined as the same number of hyperparameter trials.

Why not just…?

Why not just fine-tune a foundation depth model, which already runs at 55 FPS? This is the most serious threat on scale-aligned depth error. Surgical-DINO's AbsRel of 0.053 on SCARED may be hard to beat on that metric alone. Two things argue for the solver. PC-Depth reports that its physics-based approach outperformed DepthAnything on C3VD. And a fine-tuned network outputs neither a focal length nor a shift it can justify on a new scope. Our experiment settles the question. We stack the solver on top of the fine-tuned network and test two outcomes. The first is whether normal error and high-frequency error drop while AbsRel holds. The second is whether, on a held-out scope (Hamlyn or SERV-CT), the solver repairs the shift errors that the network makes under domain shift. If neither happens, the solver adds nothing and we will say so.

Why not just let a DUSt3R-style model infer the camera implicitly, as Endo3R and StreamSplat do? These models learn camera geometry from data and remain silent about the physics. StreamSplat reports that its orthographic simplification fails exactly in strong close-up perspective, which is the near-field regime where falloff is most informative. Our perturbed-intrinsics and held-out-scope tests compare recovered focal length directly against calibration for both approaches.

Why not just train the near-field losses into the network and skip test-time optimization? That amortized route is essentially PC-Depth extended to unknown intrinsics, and it is a credible alternative. We will build it as a distilled variant: a network supervised by our solver's outputs. We will then report the accuracy and latency gap between the distilled network and the solver. If the network closes the gap, the contribution becomes the self-calibrating loss rather than the runtime solver. That would still be a useful result, but a different one.

Proposed Experiments

Main benchmark. C3VD is the main benchmark because it combines near-field colonoscopy lighting with CT-registered ground truth. On it we report AbsRel and δ1 after median scaling, normal angular error, and error restricted to a high-pass band of the ground-truth depth. We also report focal-length and principal-point error against the dataset calibration, and the temporal variance of the recovered scale.

Generalization and latency. SCARED and SERV-CT test cross-domain transfer, using the PC-Depth protocol of training on C3VD and testing on SERV-CT. Hamlyn serves as zero-shot qualitative and latency testing on in-vivo footage. SimCol3D, with known synthetic intrinsics and lighting, is used for the identifiability study. There we can vary relief, depth range and specular strength, and check whether focal-length recovery degrades as the analysis predicts.

Decisive checks. Three checks decide the claim.

  1. Feed every method deliberately wrong intrinsics and confirm that our recovered focal length converges back within a few percent on tubular colon scenes, while acknowledging failure on flat scenes.
  2. Run the spherical-harmonic lighting ablation at an identical iteration budget. It must lose on both calibration and detail for the near-field claim to stand.
  3. Measure end-to-end latency on an RTX 3090, which must stay below 50 ms per frame including the backbone.

Downstream test. As a secondary demonstration of the calibration benefit, we replace StreamSplat's or Endo3R's input depth with refined, self-calibrated depth on C3VD sequences. We then report Chamfer distance against ground truth.