Rebalancing Rare Tissue Events in Deformable Surgical Reconstruction

Fast 3D reconstruction of moving surgical tissue now runs at hundreds of frames per second. That is fast enough, in principle, to tell a surgical robot where the tissue surface is while it moves. These models are trained and scored on whole video sequences, though, and most frames show slow, regular motion such as breathing or gentle sliding. The moments that matter for safety are rare: an instrument pressing in, tissue snapping back after release, a cut. Those frames are a small minority. A loss that averages over every frame barely sees them, and an image-quality score that averages over every pixel barely notices when they are wrong. This is the same failure that long-tailed medical classification exposed. There, ordinary training gave 1.7% accuracy on rare chest X-ray findings, and the overall average hid it.

We propose to treat rare tissue events as the "tail classes" of a reconstruction problem and to fix both how such models are scored and how they are trained. On the scoring side, we stratify frames into head, medium, and tail by how often their deformation event occurs. We then measure depth and boundary error only near the instrument, and report how often the reconstruction would wrongly signal that the tool is safely clear of tissue. On the training side, we borrow the two-stage recipe that rescued tail accuracy in chest X-ray classification. The model first learns the tissue's shape and appearance from all frames. That shape is then frozen, and only the motion model is refit with rare events sampled as often as common ones. The claim is specific: shape-frozen, event-balanced refitting of the motion model reduces tool-adjacent error on rare frames more than per-pixel motion weighting, naive oversampling, or longer training, and it costs little global image quality.

Where this sits

The reconstruction side supplies the method we build on. Endo-4DGS (Endo-4DGS) represents the scene with Gaussian splatting. In plain terms, the scene is a cloud of small 3D blobs drawn directly to the screen, and a small network moves the blobs over time. It renders at 100 frames per second. Deform3DGS (Deform3DGS) replaces that network with learnable per-blob temporal basis functions, trains in about a minute, and renders at 338.8 frames per second. HFGS (HFGS) is our closest competitor. It upweights motion-intensive regions using optical flow, which is the estimated per-pixel motion between frames. EndoWave (EndoWave) separates specular glare from tissue texture with wavelet losses. All four report only whole-image fidelity: PSNR (peak signal-to-noise ratio, a pixel-error score), SSIM (a structural similarity score), and LPIPS (a learned perceptual distance).

HFGS's emphasis works per pixel and inside every frame. It gives a fast-moving region more weight, but it does not change how often the optimizer sees the rare frames, and it says nothing about evaluation. We rebalance at the level of events and frames, and we separate learning shape from rebalancing motion. HFGS has no such separation.

The imbalance side supplies the training recipe and one constraint. The chest X-ray long-tail benchmark (Long-Tailed Classification of Thorax Diseases on Chest X-Ray) showed that freezing a representation learned on all data and retraining only the final layer with class-balanced sampling raised tail accuracy on NIH-CXR-LT from 1.7% to 30.0%. The same benchmark found that MixUp, which blends pairs of images, hurt performance because blending blurs fine local cues. That result rules out the obvious way to create extra tail data here, which would be interpolating between frames. HFGS and EndoWave independently show that smoothing high-frequency content smears deforming tissue. Our refitting therefore resamples real tail frames and never synthesizes blended ones.

The evaluation side supplies the template. The radiology report study (Evaluating Progress in Automatic Chest X-Ray Radiology Report Generation) showed that word-overlap scores miss clinically serious errors. A clinically grounded composite score, RadCliQ, reached a Kendall-tau of 0.522 against radiologist judgments. Under that score, the best report generators were only 5.5% better than retrieving a random report. We expect a similar effect: whole-image PSNR may hide that reconstruction methods are indistinguishable on the frames that matter.

The tremor side sets the constraint that gives the metric its meaning. SMAR (A soft micron accuracy robot design and clinical validation for retinal surgery) cut involuntary hand drift in patients from 299.66 µm to 41.07 µm using mechanical balancing and motion scaling. Its authors note that it has no view of the deforming tissue and list intraoperative depth feedback as future work. A robot that already removes human tremor gains nothing from a vision model whose error near the tool exceeds the error the robot has already eliminated.

So we express the error budget as a threshold sweep, and we read reconstruction error as useful only below the residual positioning error of the controller it would feed. We do not test on retinal data, because the file lists no public retinal 4D benchmark. The laparoscopic datasets operate at a coarser scale, and our budgets are scaled to them rather than borrowed from retinal numbers.

Research Program

Event taxonomy and stratified benchmark. For EndoNeRF (in-vivo prostatectomy with pulling, cutting, and tool occlusion), StereoMIS (in-vivo porcine stereo), and SCARED (porcine cadaver with ground-truth geometry), we label each frame with a deformation event. The labels are quasi-static, periodic physiological motion, tool-contact push or pull, release, and topology change such as cutting.

Labels come from instrument masks together with non-rigid optical flow left over after removing camera motion. Two annotators audit a subset, and we report their agreement. Frequency across the pooled data then defines head, medium, and tail, mirroring the chest X-ray benchmark's stratification.

Metrics. We report three. The first is tool-adjacent depth error: error inside a dilated band around the instrument mask, per stratum. The second is tool-adjacent boundary error: distance between the reconstructed and reference tissue surface edges in that band. The third is the control-relevant metric, the false-clearance rate. This is the fraction of tail frames where the reconstructed surface says the tool tip is farther from tissue than a budget $\delta$ while the reference geometry says it is closer.

We sweep $\delta$ rather than fix it. Reference depth comes from SCARED's ground truth and from stereo on StereoMIS. On EndoNeRF we use only the depth the dataset provides, and if it proves too noisy for this purpose we drop EndoNeRF depth claims.

Method. Stage one trains Deform3DGS or Endo-4DGS exactly as published. Stage two freezes the blobs' canonical positions, shapes, and colors and refits only the temporal deformation parameters. It samples frames uniformly across event classes and computes the photometric and depth losses with extra weight in the tool-adjacent band.

The motivating analogy is the chest X-ray recipe: shape plays the role of the learned representation and motion plays the role of the classifier head. Freezing shape should prevent the overfitting that naive oversampling causes, where a handful of tail frames distort the appearance model.

Baselines

The naive reference is each backbone trained as published and evaluated with our stratified metrics. Its numbers show whether the tail gap exists at all. If tail-stratum tool-adjacent error is not clearly worse than head-stratum error, the premise fails and we report that.

The strongest existing methods are HFGS and Deform3DGS. HFGS reports 38.14 PSNR on EndoNeRF (HFGS) and is the most direct competitor, because its flow-based temporal emphasis already targets fast motion. Deform3DGS reports 37.90 PSNR on EndoNeRF (Deform3DGS) and is the backbone we modify. EndoWave reports 38.93 dB on the EndoNeRF "Cutting" sequence (EndoWave), and we include it as a fidelity reference where it can be reproduced. For any of these on tail-stratified, tool-adjacent, or false-clearance metrics, no reported result was found on Lacuna, so we compute them ourselves.

All methods run on the same sequences and splits. Each gets the same number of total optimization iterations and the same hyperparameter-search budget, so stage two's extra iterations are matched. Beating the baselines means a lower tail-stratum tool-adjacent error and false-clearance rate across seeds, with a confidence interval excluding zero, while global PSNR stays within a margin fixed before the runs.

Three ablations isolate the key ingredient. Balanced sampling without freezing tests naive oversampling. Freezing plus continued uniform sampling tests whether the gain is just extra training. Our stage two with HFGS's flow weighting in place of event balancing tests per-pixel against per-event emphasis.

Why not just…?

Why not just use HFGS's flow-weighted loss? It is the real threat. Rare events often coincide with large flow, so per-pixel motion weighting may already capture most of the gain. The difference is that HFGS increases weight inside frames the optimizer samples at their natural rate, and it does not separate shape from motion. The swap ablation settles this directly. If flow weighting matches event balancing on tail metrics, we will report that the evaluation is the contribution and that HFGS already does the training part well.

Why not just oversample the hard frames? The chest X-ray benchmark and the Balanced-MixUp line of work (Mitigating Long Tailed Distributions in Medical Imaging) report that naive oversampling overfits scarce examples. In reconstruction, we expect it to bend canonical appearance toward a few frames and degrade head frames. The unfrozen balanced-sampling ablation measures this, by checking head-stratum error as well as tail-stratum error.

Why not just report per-frame errors and skip a new method? This is a fair minimal version. The stratified benchmark stands as a contribution even if no training change helps. We still test the method, because a metric that reveals a gap invites the question of whether the gap is fixable cheaply.

Why not rely on motion scaling and static virtual fixtures and skip reconstruction? Fixtures here are software boundaries that stop the tool from crossing a preset surface. Static registration fails once soft organs shift by centimeters (Automatic Patient Registration for Port Placement). In SuPer (SuPer), a static map failed after three grasps, while deformable tracking completed five. Tremor filtering removes human error but cannot follow a moving boundary.

Decisive Checks

The first check is whether the tail gap exists: naive backbones must show clearly worse tool-adjacent error on tail frames than head frames on SCARED and StereoMIS.

The second is whether the metric means something clinically. Following the RadCliQ validation logic, surgeons rank renderings of tail frames from all methods by whether they would trust the geometry for guidance. Our tail metrics should correlate with those rankings, measured by Kendall-tau, more strongly than whole-image PSNR does.

The third is whether event-balanced, shape-frozen refitting beats both the HFGS-weighting swap and the extra-iterations control on false-clearance rate across the full $\delta$ sweep.

The fourth is a replay test of closed-loop relevance. Using recorded tool trajectories, we compute when a distance-based boundary built on each reconstruction would have failed to engage before contact. Fewer missed engagements on tail frames is the outcome that would justify feeding these reconstructions to a tremor-filtered controller like SMAR.