Symmetry-Gated Completion for Feed-Forward Reconstruction Under Transient Occluders

Robots that build a 3D model of a table before grasping face a basic conflict. Fast reconstruction networks assume the scene holds still while it is photographed, but real workspaces contain people's arms, the robot's own arm, and objects passing in front of the cameras. Recent feed-forward models handle this by detecting and deleting the parts of the 3D model that belong to these moving "distractors." Feed-forward here means a network that maps a few images to a 3D model in one pass, with no per-scene optimization. Deletion leaves holes, and for a robot the holes tend to land in the worst place: the side of the target object that the distractor was covering, which is often where a gripper finger needs to go. Grasping research has long dealt with missing geometry differently. It projects a messy, partly seen object onto a simple symmetric stand-in, such as a surface of revolution or a mirror-symmetric shape, and reasons about contacts on that stand-in.

We propose putting that second idea inside the first. A feed-forward reconstructor predicts, for each object, a symmetric stand-in shape together with a confidence for each region saying whether the stand-in or the direct observation should be trusted there. Regions hidden by distractors are filled by reflecting or revolving confidently observed surface, and only where the network judges the object to be symmetric. The result is a single-pass, distractor-free 3D model whose occluded object surfaces are complete enough to grasp from. The open question we target is measurable. Does symmetry-based completion, gated per region, recover graspable geometry that distractor pruning destroys, without hallucinating wrong surfaces on asymmetric objects?

Core Claim

The gap this work addresses comes from one specific system. Distractor-free Generalizable 3D Gaussian Splatting (DGGS) is a feed-forward model that flags transient content through cross-view consistency checks and prunes the corresponding 3D Gaussians, which are small colored ellipsoids that together render the scene. Its authors state that surfaces hidden by distractors in every reference view come out as holes, and they leave generative filling to future work. Separately, SymmetryGrasp shows that predicting a symmetry plane or axis from a single clean RGB-D view lets a grasp network place contacts on the unseen "dark side" of an object. Its real-robot success is 74% on symmetric objects but falls to 47% on irregular organic ones. That drop is the evidence that symmetry is a valuable prior which must not be applied blindly.

Our claim has three parts. First, a feed-forward Gaussian reconstructor can carry a per-object "canonical proxy head" that outputs a symmetry type (none, reflective, or rotational), the symmetry parameters, and a per-Gaussian gate $g \in [0,1]$. Second, using that gate to decide whether a pruned region is completed by symmetry or left empty raises grasp success under distractors above both DGGS and cascades of existing tools. Third, the gate keeps performance on asymmetric objects at least as high as no completion. The novelty is not symmetry completion on its own; Autonomous Marker-Less Rapid Aerial Grasping already reflects partial point clouds about an estimated axis. It is a learned, visibility-aware decision of where symmetry may overwrite missing evidence, trained jointly with distractor pruning inside a single forward pass.

Research Program

Backbone. We build on a sparse-view Gaussian predictor in the style of pixelSplat, which matches features along epipolar lines across views and regresses Gaussian parameters directly. We add a distractor-pruning stage trained the way DGGS trains its own: masks come from cross-view consistency and are refined by a segmenter during training only. At inference the pruning is a learned per-Gaussian score, which avoids DGGS's runtime cost from external segmentation.

Canonical proxy head. Static Gaussians are grouped into objects, using a class-agnostic instance head trained on GraspNet-1Billion scenes. For each object, a small attention module predicts the symmetry type and parameters. Training supervision comes from the object models in GraspNet-1Billion, whose symmetries can be computed offline. For each pruned or unobserved surface region, the head mirrors or revolves nearby observed Gaussians into candidate completion Gaussians. Each candidate receives a gate value $g$, which multiplies its opacity.

Training the gate. We train the gate with a geometric target: a candidate should be kept exactly when it lies within a small tolerance of the true object surface. This makes the head learn when an object only approximately matches its proxy. It also lets the model use the proxy for a symmetric mug body while refusing to mirror the handle.

Grasping. We follow Grasp Moduli Spaces. That work maps an object to a canonical surface of revolution, finds a heuristic grasp there, and then slides contacts by gradient descent on a continuous grasp-quality measure, beating fixed-coordinate transfer in 143 of 144 cases. We first sample antipodal grasp candidates using normals from the gated proxy-plus-observation surface. We then refine the contacts locally on the completed Gaussian surface. Where the gate is low, we fall back to normals taken directly from the observed Gaussians, as in GaussianGrasper.

Training data. We composite moving occluders onto multi-view tabletop scenes, adapting DGGS's protocol of pasting MS-COCO objects onto RealEstate10K and ACID backgrounds. Here the backgrounds are the GraspNet-1Billion scenes, which contain 97,280 images over 190 scenes and therefore many viewpoints per scene. Occluders include rendered arms and hands as well as COCO cut-outs. We control how much of each target is hidden across all selected views.

Baselines

The naive reference point is a plain feed-forward reconstructor run on the corrupted views with no distractor handling. We use the pixelSplat-style backbone, and MvSplat where DGGS reports it, followed by the same grasp sampler. On the RobustNeRF "Statue" scene, MvSplat reaches a peak signal-to-noise ratio (PSNR) of 15.45 dB according to DGGS. This baseline shows how badly ghost Gaussians hurt both rendering and grasp normals.

The strongest existing method on the reconstruction side is DGGS itself, which reports 21.02 dB PSNR on "Statue." We run it with its released settings on the same views. Its pruned Gaussians go to the same grasp sampler and refinement, so the only difference from our system is completion. We beat it if grasp success and dark-side surface accuracy rise in the heavily occluded stratum, and RobustNeRF and NeRF On-the-go rendering stays within a small margin of DGGS. For On-the-go, no reported per-scene DGGS numbers were found on Lacuna, so we will rerun DGGS ourselves.

The strongest existing method on the grasping side is a cascade. We mask distractors per image with an off-the-shelf 2D segmenter, reconstruct with the same backbone, fuse to a point cloud, and run SymmetryGrasp's symmetry and antipodal heads. Those heads report grasp quality of 0.382 on seen and 0.297 on unseen GraspNet-1Billion objects and an antipodal score of 0.809. We retrain the cascade on our composited data with the same compute and tuning budget as our model.

The closest ablation is post-hoc completion. It runs DGGS, estimates symmetry on its output, and reflects every pruned region with no gate, in the spirit of Bauer et al. Two further ablations isolate our key ingredient: our full model with the gate fixed at one, and our model with the proxy head removed. A per-scene upper reference uses RobustSplat with surface-normal grasp filtering as in GaussianGrasper, which reports 85.0% real success with normal guidance versus 78.3% without. We report its minutes-scale latency alongside its accuracy. No reported result was found on Lacuna for any method's grasp success under transient distractors, so all distractor-condition grasp numbers will be produced in this work.

Why not just…?

Why not just mask distractors with a 2D segmenter and feed clean pixels to an existing pipeline? Several papers report failure modes for this approach. Split4D reports that missed or misaligned 2D masks propagate permanently into 3D. RobustNeRF notes that detectors miss transients outside their categories. NeRF On-the-go finds that segmenters miss camouflaged distractors. More important, masking does not fill the hidden surface; it only avoids ghosts. The cascade remains a real threat on symmetric, lightly occluded objects, so it is a headline baseline, and we expect to lose nothing to it only in the low-occlusion stratum.

Why not just reflect the pruned region after reconstruction? This is the most serious alternative, because it is cheap and uses the same prior. Our position is that ungated reflection is exactly what produced SymmetryGrasp's 47% on organic shapes. The post-hoc and gate-fixed-at-one ablations test this directly. If ungated post-hoc reflection matches our model on both symmetric and asymmetric objects, the integrated design adds nothing, and we will report the simpler method as the answer.

Why not just wait or take more views until the occluder leaves? In a static workspace this works. Our regime is one where waiting is not available: the occluder is the robot's own arm or a co-worker's hand during a handover, or the scene must be re-perceived between motions. We include a "more views" baseline that gives DGGS twice as many views. If doubling views closes the gap, the method's value shrinks to a latency argument, and we will say so.

Why not just use per-scene robust optimization? RobustSplat-style methods produce clean static models, but they require minutes of gradient descent per scene; even SparseGrasp needs about 4 minutes before grasping. More importantly, they still cannot recover surfaces hidden in all views. The per-scene reference tests whether completion, not optimization quality, is what limits grasping.

Why not skip 3D and grasp from a single RGB-D view? SymmetryGrasp already does this in one pass. However, a distractor between that one camera and the object leaves it with no evidence, and its reliance on clean depth is a documented weakness. We run it as the single-view baseline under the same occluders.

Proposed Experiments

Rendering. We report PSNR, structural similarity (SSIM), and the learned perceptual metric LPIPS on the RobustNeRF benchmark and the 12 NeRF On-the-go scenes. The goal is to show that completion does not degrade static-scene fidelity relative to DGGS.

Geometry and grasping. On composited GraspNet-1Billion scenes, we report three measures for the region of each target hidden in all views. These are Chamfer distance to the true object surface, the error in grasp-surface normals, and GraspNet-1Billion grasp quality. Results are stratified by occluded fraction (0–20%, 20–50%, above 50%) and by symmetry class (symmetric versus organic).

Real robot. We run a physical test following SparseGrasp's tabletop protocol of 3 views, adding a human arm or the robot arm as a persistent occluder during capture. We report success and end-to-end latency against SparseGrasp's 78.5% in clean conditions as a ceiling reference. Our target is under 200 ms from images to grasp.

Decisive check. Gains should concentrate in the high-occlusion, symmetric cell. On organic objects, our model should be no worse than DGGS without completion, while the ungated ablation should be worse. Gate calibration will be checked by plotting gate value against measured surface error. If the gate is uncalibrated, or post-hoc reflection matches us across all cells, the central claim fails.