Time-of-flight (ToF) cameras measure depth by timing how long emitted infrared light takes to bounce back. They promise what monocular endoscope video cannot give: true millimetre scale on organs during surgery. Their errors, however, are not random noise. When the returned signal is weak or saturated, the sensor's electronics add a false distance offset that depends on how bright the return is. On wet tissue, brightness depends on surface angle, moisture and blood. On a breathing or beating organ, all of these change every frame. The result is a dilemma. Today's 4D reconstruction methods treat depth input as either unbiased or scale-free. ToF self-calibration methods assume the world is rigid. Neither holds once the tissue moves, so a reconstruction pipeline fed raw ToF depth will render sensor artifacts as if they were tissue deformation.
We propose to estimate the ToF error correction jointly with a deformable Gaussian-splatting reconstruction of the organ. The correction is restricted to a tiny function of sensor-side quantities only: returned amplitude, incidence angle and range. It is shared across all frames and sequences from one sensor. Under that restriction, tissue deformation stops being a confound and becomes an asset. As the organ moves, the same patch of tissue is seen at many different amplitudes and angles, which is the excitation a calibration needs. Because the correction has around a hundred parameters rather than millions, it can be fit from the handful of sequences a surgical lab actually has.
Core Claim
The obvious combination would be to add ToF depth as a supervision term to a 4D splatting method and let a network learn a correction. It fails for a specific reason that Simultaneous Localization and Calibration (SLAC) already shows in the rigid case. SLAC jointly optimizes camera path and a static 3D lattice describing range distortion, and it works because the scene is assumed rigid. A flexible correction placed next to a flexible deformation field becomes non-identifiable: either term can explain any depth discrepancy.
Our claim is that identifiability is restored by where the correction is allowed to depend, not by how much data is available. Let $\tilde d_{p,t}$ be the ToF depth at pixel $p$ and frame $t$. Let $d_{p,t}$ and $\theta_{p,t}$ be the range and incidence angle rendered from the current 4D Gaussian model. Let $a_{p,t}$ be the measured ToF amplitude. We model
$$\tilde d_{p,t} = d_{p,t} + e_\psi\big(a_{p,t}, \cos\theta_{p,t}, d_{p,t}\big) + \epsilon_{p,t},$$
where $e_\psi$ is a uniform cubic B-spline over a coarse three-dimensional lattice with parameters $\psi$, and $\epsilon$ is residual noise. The error $e_\psi$ may not depend on pixel position, time or tissue identity. Deformation may depend on all of those, but it is also supervised by the RGB endoscope image, which carries no amplitude-dependent bias.
This yields a falsifiable prediction. Suppose a depth change correlates with an amplitude change in the same way across many tissue points and times, and the RGB images do not show matching motion. Then it is attributed to the sensor. Otherwise it is attributed to the tissue. The approach should break exactly where amplitude and true motion are perfectly collinear. We will construct that case deliberately.
Research Program
The reconstruction backbone is EndoGS-style deformable Gaussian splatting. EndoGS represents tissue as 3D Gaussians moved by a small network over factorized space-time feature planes, uses masks to ignore instruments, and adds a surface-alignment term. We reuse this as the method substrate and change only the depth term. Raw ToF depth is replaced by the error model above, with a robust loss and a curvature penalty on the B-spline lattice. The rendered surface normals supply $\theta$. To prevent the trivial solution in which deformation absorbs everything, deformation is regularized toward the motion implied by RGB photometric consistency, as in existing monocular pipelines.
The lattice design follows Lateral and Depth Calibration of PMD-Distance Sensors. That work showed that cubic B-splines fit the periodic "wiggling" range error from only 68 planar images without the boundary oscillation of polynomials, cutting distance variance from about 64 to 3.11 mm². We extend the spline's inputs from range alone to range, amplitude and angle.
The angle dependence is motivated by EVD Surgical Guidance. That paper found skin-induced ToF biases up to 7.6 mm. It corrected them with a three-parameter distance-and-angle model, but needed patient-specific parameters fitted to retro-reflective markers on a rigid skull. Markers cannot be placed on a wet, moving liver. Our model therefore lets the deformation itself supply the varying angles that the markers supplied there.
The amplitude dependence and the saturation problem are motivated by Time-of-Flight Kameras für die intraoperative Oberflächenerfassung. On static explanted porcine organs, that work handled reflective tissue by discarding pixels outside 20–80% amplitude and bilateral filtering. This still left 3–6 mm surface error.
The small-sample constraint is handled in two ways. First, the main correction is shared across all sequences from one sensor. Second, an optional low-dimensional per-session offset absorbs gross reflectance differences between tissue types, with a shrinkage prior pulling it toward zero. This mirrors the lesson of the hybrid CNN–snake ventricle segmentation: on 45 cases, a strong geometric prior beat end-to-end learning. Here the prior is the physics of where ToF errors can come from.
Data comes from two sources. The synthetic arm takes sequences with ground-truth depth and normals: C3VD from the colonoscopy 3D video dataset, and SCARED structured-light depth. We render a coaxial-illumination ToF amplitude, then inject a known error function of amplitude, angle and range, including wiggling harmonics as characterized in the ToF systematic-error investigation. The physical arm repeats the setup of Seitel et al.: a ToF camera with a co-registered RGB camera views explanted porcine lung and liver. The lung is ventilated or the liver pneumatically indented, to create controlled cyclic deformation. CT is acquired at end states for ground truth.
Baselines
The naive reference is EndoGS supervised directly with raw ToF depth. It is run with identical Gaussian initialization, iteration count and hyperparameter search budget as our method. It tells us how much damage uncorrected intensity bias does to a deformable reconstruction. We expect it to show amplitude-correlated phantom bumps on specular regions.
The classical clinical baseline is the Seitel et al. pipeline: amplitude clipping plus bilateral filtering, then the same EndoGS backbone. Its reported static ex-vivo numbers are 2.9 ± 2.1 mm mean surface distance on lung and 5.7 ± 6.7 mm on kidney against CT. We reproduce it on our deformation sequences rather than relying on those figures. Beating it means lower CT surface distance without discarding the roughly 40% of pixels it clips.
The strongest competitor on this question is offline calibration followed by reconstruction. The ToF camera is calibrated before the session with the Lindner–Kolb B-spline on a planar target, plus a Li et al.-style distance-and-angle model fitted on a tissue-mimicking phantom. The corrected depth is then fed into EndoGS. It gets the same calibration-data budget we would need to collect anyway. Beating it requires a statistically significant reduction in surface error on real tissue, paired across sequences. If it ties us, the in-situ component adds nothing.
For the monocular-prior regime, the reference is GutSee from Rendering with a Gut Feeling. It uses affine-invariant depth-prior supervision and a coaxial light model, and reports D-RMSE 4.605 mm and Chamfer distance 1.520 mm on C3VD. We run it on the same synthetic sequences to show what metric ToF buys over scale-free priors. No reported result was found on Lacuna for any ToF-input deformable reconstruction on C3VD, StereoMIS, EndoNeRF or SCARED, so all ToF-based comparisons are new runs.
Three ablations isolate the key ingredient. The first replaces $e_\psi$ with an unconstrained per-pixel, per-frame correction network. We predict it will absorb deformation, which is the SLAC non-identifiability made concrete. The second drops amplitude from the spline inputs. The third freezes deformation to rigid, giving a SLAC-like solver on moving tissue.
Why not just…?
Why not just calibrate the ToF camera once on a target and be done? This is the real threat. If a pre-session phantom calibration transfers to live tissue, our method is unnecessary. The evidence against transfer is indirect. Li et al. needed patient-specific parameters even for skin on a rigid head, and Seitel et al. saw organ-dependent saturation and bias. Reflectance shifts with moisture and blood during a session, which a pre-session fit cannot see. The offline baseline above settles this directly. We will also report the result if the gap is within noise, which would narrow the contribution to the synthetic identifiability analysis.
Why not just down-weight unreliable ToF pixels with a confidence map, as Endo-4DGS does for its priors? Endo-4DGS uses confidence-guided learning to tolerate noisy Depth-Anything depth. Confidence weighting suits random error, but intensity bias is systematic and spatially smooth. Down-weighting low-amplitude pixels removes the oblique, dark regions where depth matters most for surface shape, and still leaves mid-amplitude bias uncorrected. We include a confidence-weighted raw-ToF variant. It should beat the naive baseline but plateau above the corrected model on oblique surfaces.
Why not just skip ToF and use monocular depth priors with a scale fix? GutSee and HFGS obtain strong appearance (HFGS reports 38.14 dB PSNR on EndoNeRF) but only relative depth. Recovering metric scale needs an external reference. The C3VD comparison quantifies the metric gap. If scale-aligned GutSee matches us in D-RMSE, active sensing would not be worth its hardware cost for this task.
Proposed Experiments
The first decisive check is identifiability on synthetic data, where the injected error function is known. We measure how closely the recovered $e_\psi$ matches the truth over the observed amplitude-angle-range domain. We also measure depth RMSE and Chamfer distance of the reconstructed surface against C3VD and SCARED ground truth. The test is repeated as deformation amplitude is swept from zero (rigid) upward. The prediction is that calibration accuracy improves with deformation, because deformation spreads observations across angles. The unconstrained-network ablation should degrade instead.
A collinearity stress test makes amplitude change exactly track true motion, for example a uniformly darkening surface moving toward the camera. Here our model should fail. Its failure would confirm the mechanism rather than undermine it, and we report this regime as a stated limitation.
The second check is the real ex-vivo arm. We measure CT surface distance at deformation end states, and amplitude-binned residual bias before and after correction. The paired comparison against offline calibration is the headline result.
The third check is the small-sample learning curve. We fit $e_\psi$ from one, two, four and all sequences, then evaluate on held-out organs and a held-out tissue type. The goal is to show that a sensor-level correction fit on lung transfers to liver with only the per-session offset refit.
Throughout, we report rendering PSNR on RGB views to confirm that the depth correction does not cost appearance quality relative to EndoGS.
References
- Simultaneous Localization and Calibration: Self-Calibration of Consumer Depth Cameras
- EndoGS: Deformable Endoscopic Tissues Reconstruction with Gaussian Splatting
- Lateral and Depth Calibration of PMD-Distance Sensors
- EVD Surgical Guidance With Retro-Reflective Tool Tracking and Spatial Reconstruction Using Head-Mounted Augmented Reality Device
- Time-of-Flight Kameras für die intraoperative Oberflächenerfassung
- A Left Ventricular Segmentation Method on 3D Echocardiography Using Deep Learning and Snake
- Colonoscopy 3D video dataset with paired depth from 2D-3D registration
- A theoretical and experimental investigation of the systematic errors and statistical uncertainties of Time-Of-Flight-cameras
- Rendering with a Gut Feeling: Depth-Guided Triangle Splatting for Physically Consistent Colonoscopic Reconstruction
- Endo-4DGS: Endoscopic Monocular Scene Reconstruction with 4D Gaussian Splatting
- HFGS: 4D Gaussian Splatting with Emphasis on Spatial and Temporal High-Frequency Components for Endoscopic Scene Reconstruction