Co-Axial Specular Shading for Deformable Endoscopic Gaussian Splatting

In endoscopy the light source sits on the camera tip. As the scope moves or the tissue breathes, bright highlights slide across wet mucosa. Current deformable 3D reconstruction methods for surgical video use Gaussian splatting, which represents a scene as many small colored ellipsoids that are projected onto the image and rendered quickly. These methods assume each surface point has a fixed color. The optimizer then explains a sliding highlight the only way it can: it moves the geometry. The result is false ripples, tears, and depth ridges in tissue that never moved.

We propose to treat the highlight as information rather than noise. With the light and camera at the same point, a mirror-like reflection appears exactly where the surface faces straight back at the lens. Brightness also falls off with the square of the distance to the tip. The position of each highlight is therefore fixed by the current shape and camera pose, and no environment map is needed. We add this two-parameter physical shading term to every ellipsoid in a deformable splatting model. The highlight then constrains the surface normal instead of corrupting the deformation. Model size stays far below the cost of the generic view-dependent color that current endoscopic systems drop to save memory.

Core Claim

The claim is that a co-axial, near-light specular model removes most of the spurious deformation that moving highlights cause in 4D endoscopic reconstruction. It does so at almost no memory or speed cost. It also turns highlights into a usable cue for surface orientation on wet, low-texture tissue, where stereo matching and monocular depth priors are least reliable.

The direction that does the work is non-Lambertian reconstruction, meaning surfaces whose brightness changes with viewing angle. Non-rigid tissue deformation is where the failure occurs. The volumetric and memory bottleneck is a hard constraint: whatever we add must keep training in minutes, rendering above 25 frames per second, and per-ellipsoid parameters small.

Relation to Prior Work

Three deformable splatting systems motivate the work by exhibiting the failure. EndoGS drives per-Gaussian offsets with a small network fed by HexPlane features; HexPlanes store spatio-temporal features in a set of 2D grids. Endo-4DGS initializes from Depth-Anything pseudo-depth and reaches 100 FPS in 4 GB of memory. HFGS emphasizes high spatial and temporal frequencies. All three treat tissue as diffuse or near-diffuse, so their deformation networks bend to fit moving glints.

The SLAM systems constrain the design through their parameter budgets. EndoGSLAM drops spherical harmonics, the standard per-ellipsoid basis for view-dependent color, cutting parameters from 59 to 8. EndoFlow-SLAM also uses isotropic color. That budget tells us any appearance model must be a few parameters, not dozens.

NeRF-DS supplies the key method insight. Specular color must be computed from observation-space position and normal, not in the canonical space that dynamic models warp back to. NeRF-DS, however, assumes a distant static environment map, needs object silhouette masks, and ray-marches slowly. We keep its observation-space principle and replace the environment map with the co-axial point light, which is the correct physics inside a body cavity. That substitution removes both the map and the masks.

EndoMD-SLAM handles optical degradation with a different tool, a static-transient decomposition. It runs on rigid anatomy at 0.95 FPS. We use it as a competing explanation, not as a component.

Research Program

Each Gaussian keeps its usual canonical position, scale, rotation, opacity and diffuse albedo. A HexPlane-plus-small-network deformation field moves it over time, as in EndoGS. We add two scalars per Gaussian, a specular strength $k_s$ and a roughness $\sigma$. The normal $n$ is taken from the shortest axis of the deformed ellipsoid in observation space. Let $v$ be the unit vector from the point to the camera tip, which is also the light direction, and $r$ the distance to the tip. The color is then

$$ c = \frac{L_0\,\gamma(u)}{r^2}\Big[\rho\,(n\cdot v) + k_s \exp\!\big(-(1-n\cdot v)/\sigma^2\big)\Big], $$

where $L_0$ is a global light power, $\gamma(u)$ is a per-sequence learned radial vignetting of the light cone over the image coordinate $u$, and $\rho$ is the diffuse albedo. Because light and view coincide, the half-vector of standard specular models collapses to $v$, which is why the specular lobe depends only on $n\cdot v$. The $1/r^2$ falloff adds a weak absolute depth cue that pure photometric splatting lacks.

The program has three parts.

The first is the shading model just described, integrated into an EndoGS-style pipeline.

The second is a gradient-routing study. The specular residual can be allowed to update Gaussian rotations (normals) but blocked from updating deformation translations. We test whether this routing matters, because it is the mechanism that stops highlights from being explained as motion.

The third is a cost audit. It reports parameters per Gaussian, peak VRAM, training time and FPS against both the no-SH and full-SH variants.

Baselines

The naive reference point is EndoGS-style deformable splatting with isotropic diffuse color and no special highlight handling. This is the setting that produces the artifact. It is run with our exact code path minus the specular term, so the comparison isolates the shading model.

The strongest rendering competitor is HFGS, which reports 38.14 dB PSNR on the EndoNeRF dataset against 36.84 dB for EndoGS. The strongest geometry competitor on deforming in vivo video is FLex, which reports an L1 depth error of 1.273 mm on StereoMIS. For Endo-4DGS on StereoMIS, no reported per-sequence result was found on Lacuna. All methods are retrained by us on the same sequences with the same initialization depth source. Each gets the same iteration count and a matched hyperparameter sweep of eight configurations, all on one GPU model.

Beating these baselines does not mean higher PSNR. Baked-in highlights can render well while the geometry is wrong. Success means three things on the same benchmarks:

  1. PSNR within 0.5 dB of HFGS.
  2. A clear reduction in depth error restricted to highlight-traversed pixels.
  3. Much less spurious deformation, as defined in the experiments below.

The key ablation removes only the co-axial constraint. It replaces our two-parameter lobe with a free two-parameter view-dependent term not tied to $n\cdot v$. This tells us whether the physics or the extra capacity is doing the work.

Why not just…?

Why not just threshold and mask the highlights before reconstruction? This is the most serious threat, and it is close to what EndoGSLAM does with over-exposed pixels. On a rigid phantom with an RGB-D sensor this works. In deforming tissue, masking removes gradient signal exactly where the surface is wet and curved, and the file reports holes and smudging when masked regions lack supervision. Masks also miss the dim specular halo around each core highlight. We include a masked variant with a tuned luminance threshold and a dilation radius. If it matches our method on spurious deformation and highlight-region depth, the shading model is not needed and we will say so. Our method should win specifically on normal accuracy, because masking discards the cue we exploit.

Why not just turn spherical harmonics back on? This costs up to 48 extra parameters per Gaussian, which is the overhead the SLAM systems removed. More importantly, NeRF-DS shows that canonical-space view-dependent color cannot follow a highlight whose position depends on the deformed normal. We run degree-3 harmonics as a baseline and report both its memory cost and its spurious deformation.

Why not just absorb highlights into a per-frame transient layer, as EndoMD-SLAM does for lens debris? A transient layer can explain anything, including real motion, and adds per-view parameters that grow with sequence length. We implement a lightweight per-frame transient image as a baseline. If it removes spurious deformation equally well, the remaining advantage of our method must show up in geometry, through recovered normals and depth on highlight pixels.

Proposed Experiments

The decisive test uses rigid sequences from C3VD. The colon phantom does not move, but the scope does, so highlights slide while the true deformation is zero. We run every deformable method there and measure the magnitude of the predicted deformation field. We also measure its spatial correlation with tracked highlight trajectories. A good model should predict near-zero motion. For reference, EndoFlow-SLAM reports a depth RMSE of 1.54 mm on C3VD. We additionally score depth and normals against C3VD's ground truth, separately for highlight-traversed and non-highlight pixels.

The second test covers real deformation. We use the C3VDv2 sequences with blood-pressure-cuff deformation and in vivo StereoMIS, where geometry is scored against stereo-derived depth as FLex does. Here we check that the specular term does not suppress real motion.

The third test checks that rendering quality holds on the EndoNeRF dataset, against Forplane's 37.31 dB and HFGS's 38.14 dB.

Two controlled checks complete the study. The first is a synthetic sweep that renders known deforming meshes under a co-axial light. It varies specular strength and checks whether recovered $k_s$ and normal error track the ground truth. The second is the cost audit: our method should stay within roughly 25% of the diffuse baseline's parameter count, VRAM and FPS.