Fast methods for reconstructing moving, deforming tissue from endoscope video store the scene's motion in a few flat 2D feature grids, one for each pair of axes among x, y, z and time. To find out how a tissue point moves at a given moment, the method reads these grids at the point's projected coordinates by blending the four nearest cells (bilinear interpolation). This read is smooth by construction. Surgical motion is not. Where a tissue flap is lifted off the organ beneath it, where a cut opens, or where retracted tissue slides past tissue that stays put, the true motion jumps sharply across a thin boundary. A smooth read blends the two sides, and the renderer shows the "melted" edges and ghosting that recent work reports.
We propose to replace the fixed four-cell read with a learned, content-dependent read. For each query, a small predictor chooses a handful of nearby sample locations on the plane and weights for them, so samples can stay on the correct side of a motion boundary instead of straddling it. The idea borrows the mechanism of deformable convolution, which learns where a filter looks, and applies it to the feature-plane lookup of dynamic Gaussian splatting. The central claim is falsifiable and narrow. Much of the remaining error in these reconstructions sits at motion discontinuities, and that error comes from interpolating across them. It is not mainly a shortage of grid resolution, and it is not mainly a shortage of high-frequency supervision.
Core Claim and Positioning
The strongest surgical reconstruction systems all store deformation in axis-aligned planes that are read by rigid bilinear lookup. Forplane splits 4D space-time into three spatial and three space-time planes and ray-marches through them. EndoGaussian uses a similar six-plane "HexPlane" to decode per-Gaussian translation, rotation and scale, and renders at 195 FPS. DMiT makes tri-planes multi-scale through mipmapping, but it does so on generic dynamic scenes and keeps each level a uniform grid read by the same smooth lookup. None of these changes where or how the planes are sampled.
The work closest in spirit is HFGS. It diagnoses the same blurry, under-reconstructed edges during fast deformation and treats them with training losses: a Fourier high-pass image loss, plus optical-flow priors that regularize the deformation field. HFGS changes what the model is penalized for. We change what the deformation representation is able to express.
On the sampling side, three works supply the technique and the motivation. Deformable Convolutional Networks and Deformable ConvNets V2 provide the learned-offset, learned-weight sampling operator we reuse. DefGrid supplies motivating evidence that aligning the sampling grid to boundaries beats adding resolution: at 1/8 downsampling it reaches 65.36% mIoU on Cityscapes against 59.45% for fixed pooling. We do not borrow DefGrid's mesh. Its fixed triangle topology cannot represent cuts, which is exactly the case we care about. Instead we deform a point-wise read, and that read has no topology to break.
Research Program
The first step is a diagnosis, and the project depends on it. We train EndoGaussian and HFGS on EndoNeRF sequences with cutting and pulling. We then bin rendering error by distance to motion boundaries, where a boundary is defined by the spatial gradient of pretrained optical flow between consecutive frames. HFGS already uses the same flow source for its temporal prior. If error is not concentrated in thin bands near high flow gradient, the core claim fails, and we stop before building anything.
If the diagnosis holds, we build the method. It keeps EndoGaussian's structure: Gaussians initialized from monocular depth, and a HexPlane deformation decoder. It changes only the lookup on the three space-time planes, and optionally on the spatial planes. For a query coordinate on a plane, a first bilinear read gives a coarse feature. A tiny linear head maps that feature to $K$ offsets (with $K$ = 4 or 8) and softmax weights. The final feature is the weighted sum of bilinear reads at the offset locations. Offsets are bounded to a few cells so the read stays local.
This read happens once per Gaussian per timestamp, not once per pixel. The cost therefore scales with the number of Gaussians, and we will measure it directly rather than assume it is small. The concern about irregular memory access raised by Chang et al. is real, and frame rate is reported alongside every quality number. Initializing the offset head so that it reproduces plain bilinear lookup lets training start from the baseline's behavior.
The last step is analysis. We visualize the learned offsets to check whether they point away from motion boundaries or behave as generic blur. We also test whether the gains concentrate in the boundary bands identified in the diagnosis.
Baselines
The naive reference point is EndoGaussian unchanged, with rigid bilinear HexPlane lookup, rerun with our code on our splits. EndoGaussian reports 37.849 PSNR on EndoNeRF. Because our method is a drop-in change to this system, beating our own rerun under identical Gaussian counts, iterations and learning-rate search is the primary test. The paper's number serves only as a sanity check on the reproduction.
The strongest existing method is HFGS, which reports 38.14 PSNR on EndoNeRF, the best figure in the file. We run it on the same sequences, with the same hardware and the same number of hyperparameter trials. We also run it with our lookup added, since the two ideas can combine. Beating it means one of two outcomes: a higher boundary-band PSNR with overall PSNR no worse, or matching its quality without its losses at a frame rate still above 60 FPS. No reported result was found on Lacuna for a boundary-band metric on any method, so every system gets this number from our own runs.
The ablation that isolates the key ingredient keeps the $K$-sample read but freezes the offsets to a fixed, content-independent stencil of the same radius. This version adds the same compute and the same smoothing footprint but cannot adapt. If it matches the learned read, the benefit comes from a wider stencil and not from boundary awareness. A second ablation applies learned offsets only to the spatial planes or only to the space-time planes, to test our assumption that the problem lives in how motion is stored.
Why not just…?
Why not just raise the plane resolution? A higher-resolution grid shrinks the blended band at a discontinuity but never removes it. Memory also grows with the plane area: FastEndoNeRF had to cut dense representations from 12GB to 4GB to be usable. This alternative is a genuine threat, so we plot boundary-band error against GPU memory for rigid planes at two to four times the base resolution, and separately for DMiT-style mipmapped planes. The proposal survives only if the learned read reaches, at base resolution, the boundary error that rigid planes need substantially more memory to reach.
Why not just add HFGS-style high-frequency losses? A loss can demand sharp boundaries, but a smooth lookup may be unable to produce them without very fine grids. We expect HFGS's own gains to concentrate away from motion boundaries, and the diagnostic binning will show whether that is true. If HFGS alone already clears the boundary bands, our capacity argument is wrong and the result is reported as negative.
Why not just drop the grid and use an MLP deformation field, as Endo-4DGS does? MLPs tend to fit smooth functions first, which also blurs discontinuities, and Endo-4DGS is reported to struggle with sudden tool-tissue contact. We include it as a baseline and bin its error in the same way.
Why not just add a smooth coarse pre-warp before the planes? A smooth pre-warp cannot create a jump. We include one anyway, because if our learned offsets end up smooth in practice, this simpler design would be equivalent and preferable.
Proposed Experiments and Decisive Checks
The main benchmark is EndoNeRF, which contains prostatectomy footage with cutting and pulling. We report PSNR, SSIM, LPIPS, boundary-band PSNR at several band widths, FPS and memory. SCARED, which has structured-light ground truth, tests whether sharper renders reflect correct geometry. There we report depth error against the reference scans, since EndoSurf reports 0.952mm point-cloud distance on SCARED as a geometric reference. StereoMIS provides a transfer check. Every configuration runs with three seeds.
The decisive outcomes are fourfold. The diagnosis must show that error concentrates at motion boundaries. The learned read must beat both the frozen-stencil ablation and the resolution-matched-memory rigid planes in the boundary bands. The gain must persist when combined with HFGS. Frame rate must stay above real-time. If the frozen stencil matches the learned read, or if extra resolution matches it at comparable memory, we will conclude that grid lock is not the bottleneck in surgical splatting, and that conclusion is itself useful to report.
References
- Efficient Deformable Tissue Reconstruction via Orthogonal Neural Plane
- EndoGaussian: Gaussian Splatting for Deformable Surgical Scene Reconstruction
- DMiT: Deformable Mipmapped Tri-Plane Representation for Dynamic Scenes
- HFGS: 4D Gaussian Splatting with Emphasis on Spatial and Temporal High-Frequency Components for Endoscopic Scene Reconstruction
- Deformable Convolutional Networks
- Deformable ConvNets V2: More Deformable, Better Results
- Beyond Fixed Grid: Learning Geometric Image Representation with a Deformable Grid
- An Efficient Accelerator Design Methodology For Deformable Convolutional Networks
- Efficient EndoNeRF reconstruction and its application for data-driven surgical simulation
- Endo-4DGS: Endoscopic Monocular Scene Reconstruction with 4D Gaussian Splatting
- EndoSurf: Neural Surface Reconstruction of Deformable Tissues with Stereo Endoscope Videos