Seeing the Part No Camera Saw: Visibility-Gated Fusion for Semantic Shape Completion

Robots and cars sense the world through two imperfect channels. Depth sensors such as LiDAR give exact 3D positions but only for surfaces facing the sensor, so every object arrives as a hollow, partial shell. Cameras give rich information about what things are, but only for pixels that were actually seen. The common way to combine them is to project each 3D point into the image and copy the image's features or labels onto it. This leaves two failures that nobody fixes together. Points on a hidden background surface project onto the foreground object in front of them and inherit its label. The back half of every object, which no sensor observed, gets no meaning at all. This note proposes one model that does both jobs. A learned shape prior, meaning a network trained on many complete shapes that can guess the missing geometry of a partial one, first completes each object. The completed shape is then used for two things. It decides which points each pixel can truly see, and it carries meaning from the seen part of an object to the unseen part.

The core claim is that labeling geometry no camera observed is a distinct problem that neither image–point fusion nor shape completion solves alone, and that solving it requires an explicit split between visible and hidden points. Hidden-part semantics should be predicted only from features that were lifted under a correct visibility test, because otherwise the predictor learns from occlusion-corrupted inputs. We expect the gains to sit almost entirely on hidden points and on points near occlusion boundaries. Most of the work is in evaluating those two regions separately, since averaged scene metrics would hide them.

What the literature gives and leaves open

The fusion literature supplies the lifting step and also motivates the problem. PointFusion projects each point into the image with the camera calibration and concatenates the local image feature with the coordinates. MVX-Net does the same at point and voxel level before 3D convolution. 3DMV backprojects features from several views into a shared voxel grid. SGFormer replaces concatenation with cross-attention, in which 3D points query image tokens. All four attach image evidence only to points that were measured, and none tests whether a point was actually visible in the pixel it samples. The research file names this "occlusion parallax" as an open failure of direct projection. Countering Multi-modal Representation Collapse through Rank-targeted Fusion constrains our design. It shows that fused networks tend to rely only on the denser image channel, so our hidden-point predictor must be forced to use geometry. Otherwise it could learn to copy the nearest visible label.

The completion and prior literature supplies the geometry. PCN encodes a partial cloud into a global feature, then decodes a coarse skeleton and folds 2D grids into fine surface patches. DeepSDF represents shapes as the zero level of a learned signed-distance network and fits a latent code to partial input at test time. Unified Primitive Proxies for Structured Shape Completion decodes primitive surfaces alongside dense points. All three output geometry without semantics, and none consumes image evidence.

The closest attempts to join the two halves come from mapping. SLAM++ (Salas-Moreno et al.) retrieves a whole CAD model once a category is detected, which labels hidden parts only for objects that exist in a database. Parkhiya-style deformation subspaces fit category shape spaces during monocular mapping. GSFF-SLAM stores semantic features inside 3D Gaussian primitives, and IRIS-SLAM fuses instance representations with geometry foundation models. The feature-field systems label only what was rendered from observed views. The CAD-retrieval systems label hidden parts by swapping in a template rather than by completing the observed instance. Our proposal differs from both. It completes the specific observed instance, gates the image evidence by visibility against that completion, and predicts per-point semantics, part labels and categories, on the unobserved portion.

Research Program

The model has three stages, trained end to end. The first stage is a completion network that takes the partial point cloud plus image features of the detected object region and outputs a dense completed cloud. We use a PCN-style coarse-to-fine decoder, so that each completed point carries a flag for whether it was observed or synthesized. It also carries a predicted occupancy confidence.

The second stage is the visibility gate. We render the completed geometry into each camera as a soft depth map. A splat of each point gives a depth value weighted by its confidence. A point $p$ receives the image feature at its projected pixel with weight $w(p)=\sigma((d_{\text{surf}}(u,v)-d(p)+\tau)/\epsilon)$. Here $d(p)$ is the point's camera depth, $d_{\text{surf}}(u,v)$ is the rendered depth of the nearest completed surface at pixel $(u,v)$, $\tau$ is a small tolerance, $\epsilon$ is a temperature, and $\sigma$ is the logistic function. Completed occluders, such as the full body of a sparsely scanned pedestrian or the legs of a chair, now block the points behind them, even where raw LiDAR has holes.

The third stage is hidden-part propagation. Points with low total visibility weight across views get semantics from a small transformer. It attends over the gated features of that object's visible points and over the completion network's latent shape code. Through the latent code, category-level structure enters: the shape network knows what kind of part tends to sit where the evidence is missing.

To counter the modality collapse identified by the rank-targeted fusion paper, we train with image-feature dropout on visible points. This keeps the latent code informative. We add one auxiliary loss on hidden points that uses geometry alone.

The analysis focuses on where errors occur. We bin every evaluation point by its ground-truth visibility in the input views (visible, boundary, hidden) and by the sparsity of the raw scan. Every number is reported per bin.

Baselines

The naive reference is the decoupled pipeline the file describes. It runs a 2D segmentation network, projects labels onto observed points, completes geometry with PCN, and gives each synthesized point the label of its nearest observed point. The method must beat this on hidden points. If it only matches it there, the shape prior is contributing nothing beyond spatial smoothness.

The strongest fusion competitor is SGFormer-style cross-attention, applied to the PCN-completed cloud. Every completed point, observed or not, is allowed to query image tokens. This is the strongest competitor because attention can in principle learn to down-weight parallax-corrupted tokens on its own. It will use the same completion backbone, the same image encoder, the same training data, and the same hyperparameter search budget as our model. A 3DMV-style multi-view voxel backprojection run on the same completed geometry serves as the multi-view reference. A CAD-retrieval baseline in the spirit of SLAM++ assigns part labels from the nearest training shape after alignment.

We found no reported result on Lacuna for any of these methods on hidden-region semantic accuracy, and none on the benchmarks below. All baseline numbers will therefore be produced by us under the shared protocol. Beating the strongest baseline means a statistically reliable gain in hidden-bin mean intersection-over-union across three seeds, together with no loss on the visible bin.

Two ablations isolate the key ingredients. The first keeps everything fixed but runs the visibility gate against raw observed depth, and separately against an image-guided depth-completion map, instead of the completed 3D shape. The second removes the latent shape code from hidden-part propagation, leaving attention over visible points alone.

Why not just…?

Why not just let cross-attention learn visibility implicitly? Attention sees a corrupted token and a correct one with nothing to tell them apart except learned appearance. Occlusion parallax is a geometric fact that attention has no direct input for. The SGFormer-on-completion baseline tests this directly. If it matches our model in the boundary bin, explicit gating is unnecessary and we will say so.

Why not just z-buffer against a depth-completed image? This is the real threat for the visibility step, because image-guided depth completion fills the front surfaces of occluders, and front surfaces are all a visibility test needs. We do not argue around it. The first ablation runs exactly this, and we expect it may tie on boundary points for dense, near-range objects. Our main claim does not rest on the gate alone. It rests on hidden-part semantics, which a 2D depth map cannot provide, because the back of an object has no pixels. If depth completion ties on visibility, the final system will use it, and the contribution narrows to the propagation stage.

Why not just complete with PCN and label by nearest neighbor? That is the naive baseline. It fails whenever the hidden part has a different label from the closest visible part, for example the unseen front legs behind a visible backrest, which is the case SLAM++ was built for. The hidden-bin comparison measures the gap. A per-part breakdown will show whether the gains come from these label changes or from general smoothing.

Why not just retrieve a CAD template? Retrieval labels hidden parts well for in-database shapes and poorly for novel instances. We evaluate it on held-out shape instances and categories, where completion of the observed instance should help.

Proposed Experiments

We use three settings drawn from the file. The first is a synthetic object-level setting built from the PCN completion benchmark's shapes, rendered with RGB views and part labels, so that complete ground-truth semantics exist for hidden geometry. The second is an indoor setting from SUN3D. There, the full multi-view reconstruction with object labels is the ground truth, and a single held-in view (one frame of depth plus RGB) is the input, so hidden regions are real occlusions. The third is an outdoor setting on the KITTI data of Geiger et al.. Ground truth comes from aggregating sweeps along the trajectory, and single-sweep LiDAR plus one camera image is the input. Metrics are per-bin mean intersection-over-union for semantics and Chamfer distance for geometry.

Three outcomes would decide the question. The first is whether the model beats SGFormer-on-completion and nearest-neighbor propagation in the hidden bin on all three settings. The second is whether completed-geometry gating beats raw-depth gating in sparse, far-range KITTI bins, and whether it beats depth-completion gating anywhere at all. The third is whether removing the shape latent collapses hidden-bin accuracy back to the nearest-neighbor level. If the first holds and the third shows a clear drop, the claim stands even if depth completion wins the second.